The web is home to massive amounts of data, with more being created every day. Organizations can harness this constant stream of information to gain understanding, plan strategies, and find opportunities. Enriched news data can help your application make dynamic connections across current events faster. In this exercise, you'll start with the basics and build your own news mining web application using JavaScript, Node.js, and the Watson Discovery service. In this exercise:

• Code is written in Node.js, with the server-side using the Express framework and the client using ReactJS.
• You'll use the pre-built Watson Discovery News collection
• You'll access the Watson Discovery service through the Watson Discovery API

Optionally, you can choose to:

• Use a Slack interface to query the data
• Push news alerts out to an RSS reader
• Host the app on the IBM Cloud

This code pattern shows you how to tap into massive data sets to mine insight. You'll build a news mining web application with the Watson Discovery service using the Watson Node.js SDK. The app demonstrates two use cases using Watson Discovery News:

• Search: Query for the most relevant new articles about a specific topic or subject. Because the news collection is pre-enriched with natural language processing, you can query not just on keywords or categories but also on concepts, sentiment, and relations to get richer search responses.

• Trending topics in the news: Identify popular topics over the past 24 hours. Topics can be general, or specific to an industry or category.

1. The user interacts with the Watson Discovery News Server via the app UI.
2. User input is processed and routed to the Watson Discovery News Server.
3. The Watson Discovery News Server sends user requests to the Watson Discovery Service.
4. The Watson Discovery Service queries the Watson News Collection.
5. The Watson Discovery Service sends news articles to the RSS Reader.
6. The Watson Discovery Service responds to Slack search requests.

• Watson Discovery: A cognitive search and content analytics engine for applications to identify patterns, trends, and actionable insights.

• Node.js: An asynchronous event driven JavaScript runtime, designed to build scalable applications
• React: Javascript library for building User Interfaces
• Express: A popular and minimalistic web framework for creating API and Web server
• RSS: RSS (Rich Site Summary) is a format for delivering regularly changing web content in our case it will be trending topics
• Slack: Slack is a cloud-based set of team collaboration tools and services with chat bot integration
• Botkit: Framework for creating and managing chat bots

Click on one of the options below for instructions on deploying the app.

• Error deploying to IBM Cloud using the Deploy to Cloud Foundry option

If you see the following in the logs for a Deploy to IBM Cloud using Cloud Foundry in the Stage History, click the Redeploy button. This should get past the race condition during the deployment.

Binding service discovery-news-service to app sda-watson-discovery-news-9-4 in org scott.dangelo / space dev as scott.dangelo@ibm.com...
FAILED
Could not bind to service discovery-news-service
Error: Server error, status code: 409, error code: 60016, message: An operation for service instance discovery-news-service is in progress.

Finished: FAILED

The trending page:

The query page:

Here is a series of short videos that go into more of the details about the features of this code pattern.

• Part 1 - Navigating the UI
• Part 2 - Code walkthrough
• Part 3 - Watson Discovery service APIs

Now that you have built the base application, here are some additional steps you can take to add a Slack interface, and/or provide a way to send new trending news topics to your favorite RSS feed.

To integrate a new Slack Bot into your existing Slack team, navigate to https://<my.slack.com>/apps/manage/custom-integrations, where <my.slack.com> is the Slack workspace you want to customize.

1. From the Cutsom Integrations page, select the Bots option.

2, To add a new bot, select the Add Configuration button.

3. Enter a username for the bot and click Add bot integration.

4. Once created, save the API Token that is generated.

1. Edit the .env file and enter the Slack Bot API Token saved in the previous step.

# Slack
SLACK_BOT_TOKEN=<slack_bot_token>

2. Restart the application.

If you used the Deploy to IBM Cloud option, most of the setup is automatic, but not quite all of it. We have to update a few environment variables.

1. From the IBM Cloud dashboard find the app that was created. Click on the Runtime tab and navigate to the Environment variables tab.

2. Add and set the Slack Bot API Token saved in the previous step.

3. Save the new value and restart the application. Watch the logs for errors.

Tip: 💡 Try restaging your application if you see an error.

The slack bot will respond to certain key words, below is a sample dialog. Remember to @ the bot each time, or start a private chat. Make sure to invite your bot into other channels using /invite @<my bot>.

user: @newsbot hi
newsbot: Hello.

user: @newsbot news please
newsbot: Hi there! What news are you interested in?

user: @newsbot toronto raptors
newsbot: You want me to search for news articles about `toronto raptors`?

user: @newsbot yes
newsbot: OK searching...

On the Trending News Panel, click the RSS Feed button to launch a new tab in your browser. Cut and paste the tab URL into your favorite RSS Reader. If your RSS Feed Reader supports push notifications you can get alerted when trending topics change along with a news article for that topic.

Since RSS feed is a standard way to consume constantly changing data such as news, we can use the RSS feeds we generated to also post news articles to your organizations slack channel to track trends in your industry, or consume the feed to generate a daily digest of news and email in the morning. Other uses may include automatically posting tweets to a Twitter account on news articles on trending topics using a service called IFTTT.

• Demo on Youtube: Watch the video.
• Watson Node.js SDK: Download the Watson Node SDK.
• Cognitive discovery architecture: Learn how this code pattern fits into the Cognitive discovery Reference Architecture.
• Overview of the Watson Discovery service: Extract value from unstructured data by converting, normalizing, enriching it.
• Three ways IBM has evolved Alchemy Data News into Watson Discovery News and made it even better: Watson Discovery News takes big steps forward in NLP-enriched news search.
• IBM launches Watson Discovery Service for big data analytics at scale: Suite of APIs makes it easier for companies to ingest and analyze their data, even if they don’t have advanced degrees in data science.
• SlideShare: Building with Watson – Network Visualizations using Watson Discovery: See an app built on the Watson Discovery Service and D3.js that helps you explore your data using a network map built on NLP metadata.

• Artificial Intelligence Code Patterns: Enjoyed this Code Pattern? Check out our other AI Code Patterns.
• AI and Data Code Pattern Playlist: Bookmark our playlist with all of our Code Pattern videos
• With Watson: Want to take your Watson app to the next level? Looking to utilize Watson Brand assets? Join the With Watson program to leverage exclusive brand, marketing, and tech resources to amplify and accelerate your Watson embedded commercial solution.

This code pattern is licensed under the Apache License, Version 2. Separate third-party code objects invoked within this code pattern are licensed by their respective providers pursuant to their own separate licenses. Contributions are subject to the Developer Certificate of Origin, Version 1.1 and the Apache License, Version 2.

Apache License FAQ