AI and Machine Learning Methods

Materials 

• Computer (shared by two students)
• Cardboard Template
• Scotch Tape
• Scissors
• Ruler
• Googly Eyes 
• Arduino Leonardo
• Micro Servo (SG90)
• Five jumper wires (Male to Male)
• Two paper cups
• Two different types of dry cereal

 Preparation

1. Organize materials to be distributed to students.
2. Gather cardstock template for the students to use to build their smart sorter – cut out/assemble a sample for advance viewing.
3. Organize computers and Arduino Leonardo.
4. Try out the following site so you are familiar with the tasks: https://teachablemachine.withgoogle.com/

Design Challenge

You have been given the challenge of working as part of a team of engineers use machine learning to train your computer to distinguish between two different types of dry cereal and then accurately sort the two into separate cups via a simple device attached to an Arduino.  You’ll then re-engineer the device to improve how efficient it works.

Criteria

• The computer must be well trained or it will make errors. 
• Cereal must be sorted correctly every time.
• Students can alter the design or the code if they have ideas for improvement.

Constraints

• Time
• Materials
• Teamwork 

In this activity, we explore artificial intelligence and machine learning. Students work with the Teachable Machine to train their computer to differentiate items, verbal phrases and even human poses or motions. They develop a model using motion identification machine learning to solve a problem. They learn about supervised and unsupervised machine learning, and then program an Arduino to work with Teachable Machine models to turn a laptop into an intelligent and efficient sorting machine. Teams test their machines and strive for 100% accuracy by revising training or machine setup.

• Number of students on Team: 2
• Length of Activity: 2 hours

Advance Preparation

1. Organize materials to be distributed to students.
2. Gather cardstock template for the students to use to build their smart sorter – cut out/assemble a sample for advance viewing.
3. Organize computers and Arduino Leonardo.
4. Try out the following site so you are familiar with the tasks: https://teachablemachine.withgoogle.com/

Materials

• Student Resource and Worksheets
• Student Team Materials
  • Computer (shared among two students)
  • Two items for Teachable Machine training (coins, key, pen, pencil, etc.)
  • Cardboard Template
  • Scotch Tape
  • Scissors
  • Ruler
  • Googly Eyes
  • Arduino Leonardo
  • Micro-USB Cable
  • Micro Servo (SG90)
  • Five jumper wires (Male to Male)
  • Two bowls
  • Two different dry cereals

 Day of Activity

1. Discuss basic machine learning techniques including the differences between supervised and unsupervised machine learning and lead a discussion about the importance of both good quality and high quantity of data input (in this case photo samples). Discuss applications for machine learning such as efficient online searching and language training.
2. Explain that students will be working in teams of 2 (sharing a computer) to develop machine learning modules using the Teachable Machine. They’ll learn how to best prep the machine to make better informed decisions.
3. Using Teachable Machine, students will train computers to identify objects, human positioning, and sounds.
4. Next, students will develop a sorting machine using Teachable Machine to recognize and sort two different types of cereal.
5. Once the team has succeeded in basic cereal sorting, they will consider redesign options for the machine to work more efficiently. They may adjust code, the cardboard design, or both.

What is Artificial Intelligence and Machine Learning?

Artificial Intelligence (AI) aims to imitate human or “natural” intelligence by use of modern algorithms and datasets that imitate human learning. Practically, AI helps a machine to learn and perform actions on its own – and improve the accuracy of its recommendations and decisions over time.

One familiar application of AI is a search engine, which recommends websites to users based on individual requests, and tracks how well the suggested sites meet the users’ needs.  For example, if you search for “history of chess” the underlying program will suggest websites whose contents match the searched phrase and its individual words. Furthermore, if many users with similar search terms click on the same website but click off again quickly, the algorithm will “learn” over time that the suggested site isn’t a good match for the search criteria, and will recommend it less frequently.

Music streaming services also use AI algorithms to determine which song a subscriber might want to hear next.  In this case there may not be any explicit search criteria, and the service must learn your preferences from your actions on the platform and from any personal information it can access about you. Self-driving cars require AI algorithms to interpret the motion of other vehicles from real-time cameras, to read and understand road signs, and to make choices regarding steering, speed, and safety. AI is also well-established in gaming, where non-player characters (NPCs) and “bot” opponents must mimic human decision-making skills, often at a precisely chosen level of intelligence. Online advertising is personalized using data-driven AI, in which the search history, personal details, and social network of a user determine not only what advertisements they will receive, but when, where, and in what format they will receive them.

Machine Learning

Machine Learning (ML) is a widespread and influential variety of AI which enables machines to perform complex tasks without specific instructions or programming. This goal is achieved using statistical optimization, which sometimes incorporates feedback from the environment in which an algorithm operates, and is inspired by the immersive way that humans and other animals are able to learn. We will be working with several machine learning applications in this activity, all involving the identifications of similarities and differences among input data.

ML algorithms can be thought of as intelligent strategies for pattern recognition. Two key categories are supervised learning, where an algorithm is “trained” on example data with known properties before being applied to unknown data, and unsupervised learning, where an algorithm searches for structure and similarity in an entirely unknown dataset. A simple example of supervised learning is “classification,” where each input datapoint is assigned to one of several categories.  For example, a car moving at 60 MPH on a highway might be labelled as “normal” while another moving at 90 MPH is labelled as “fast,” alerting a self-driving car to a potentially hazardous situation and triggering a reaction. Another example of supervised learning is “regression,” where an algorithm is trained to predict one or more continuous quantities, such as forecasting temperature, rainfall, and humidity at a given location using a combination of historical data and physical weather-modeling. Unsupervised learning includes “clustering” algorithms which identify natural subdivisions of a collection data, and is widely used in computational biology, market research, and the social sciences.

Image recognition and searching for similar images (e.g. Google Lens) are successful arenas for supervised ML. If you upload a photo of a canoe, for example, many others have likely already done the same, so the algorithm will recognize your image as a canoe. This is possible either because prior users have labelled their canoe images with the word “canoe,” or because the algorithm has scraped contextual information from websites or social media posts where canoe images appear. Over time, as the algorithm absorbs more samples, the system will become more accurate and may learn to distinguish subcategories of canoe, such as styles or manufacturers. Note that the algorithm does not understand what a canoe is or how it works, but has learned to distinguish patterns of pixels which look like a canoe from those that do not. Applications of image recognition include facial or eye recognition in a security system, quality assurance in manufacturing settings, and automated diagnosis from medical scans.

Computer vision more broadly describes applications where a machine tries to understand some environment through visual data. This often involves the use of multiple AI/ML algorithms in succession, such as unsupervised learning to identify unusual or interesting regions of an image, processing steps which filter unwanted noise or improve contrast, and supervised learning to label objects or behaviors contained in the processed image. Computer vision is used to inform autonomous vehicles about their surroundings, to perform face tagging online, and to automate military combat including missile guidance.

Natural language processing (NLP) applies ML to recognize, analyze, and interpret human communication – whether in digital text, speech, or handwriting. Supervised learning is frequently used to digitize handwritten or spoken samples, using many training examples which have been tagged by a human or another algorithm. Applications of this technology include virtual assistants, transcriptions of books and letters, machine translation, and live captioning of TV shows or video meetings. Unsupervised learning is also valuable in modern linguistics, identifying patterns in language with greater speed or insight than possible for a human.

Deep learning is an important class of ML inspired by the communication systems inside animal brains.  Deep neural networks include “hidden layers” of trained algorithms between input and output data, allowing for the modeling of more complex relationships, and have shown promise in computer vision, NLP, and drug development.

Recently, ethical challenges have emerged related to AI and ML, including racial/gender biases in recognition/classification systems, safety risks from autonomous vehicles, the safety of weapons systems, and privacy concerns associated with personal data. In most cases, it is not the intelligent algorithm itself which leads to ethical issues, but instead the methods by which input data are obtained and results are interpreted or used.

Find out more at IBM Machine Learning: www.ibm.com/cloud/learn/machine-learning

• Engineers: Inventors and problem solvers of the world. Twenty five major specialties are recognized in engineering (see infographic).
• Engineering Design Process: Process engineers use to solve problems. 
• Criteria: conditions that the design must satisfy like its overall size, etc.
• Constraints: limitations with material, time, size of team, etc.
• Prototype: A working model of your solution to be tested.
• Iteration: Test & redesign is one iteration. Repeat (multiple iterations)
• Artificial Intelligence: aims to imitate human or “natural” intelligence by use of modern algorithms and datasets that imitate human learning. 
• Machine Learning: the use of algorithms used to support intelligent strategies for pattern recognition.

Dig Deeper Challenge – Data Tracking and Predictive Marketing:

In this activity, we have been considering machine learning with images and sounds, but machine learning using data (tracking browser or app usage, credit card spending habits, or voting patterns) is a broader application.

Consider how computers can evaluate large sets of data to predict outcomes. As an example, Spotify analyzes data from all users to predict what song a customer might like to hear next. The service can evaluate how much a user likes or doesn’t like a song by how long they listen to it, if they listen again, if they share it, if mark as a favorite. If the system is frequently incorrect in predicting the likes and dislikes of their customers, users will be annoyed and not use the app. On the other hand, if they consistently like the next song offered, they might use the app more frequently….which is good for advertisers and subscription renewals. Over time, this information also becomes valuable to record companies looking to produce music that will sell well and be enjoyed by the masses. However, the data needs to be accurate, and the predictions based on it must be flexible and change over time as the underlying data or usage changes.

• Can you think of other examples of machine learning and AI applications where user data can impact an industry or future products or services?
• Do you think it is an invasion of privacy for companies to use your data to inform other products or services? Did you think you agreed to the usage of your data when you downloaded your most recent new app?
• Website usage is tracked by Google Analytics so content developers can determine what website sections to expand or delete. Data is gathered on popular pages, how long a user stays on a website before going elsewhere, where the user lives, and what they searched for. Do you feel that the use of this data is reasonable? A good thing? Does it raise questions?

Dig Deeper Challenge – AI Ethics Challenges:

As AI applications expand into the realms of manipulated images, videos, and sound bites, questions may arise about putting restrictions on machine learning.

Consider this quote: “AI doesn’t have to be evil to destroy humanity – if AI has a goal and humanity just happens in the way, it will destroy humanity as a matter of course without even thinking about it, no hard feelings.” – Elon Musk

• Do you think this video overreacts to the impact of AI in the future?
• Do you think that the development of artificial intelligence puts humanity at risk?
• Are there certain applications you think should be in accessible to “intelligent machines?” Consider warfare, weather forecasting, remote surgeries — can/should all of these be managed by intelligent machines?
• Where do you think machine learning can have the greatest impact to help society? Harm society?
• Do you think that machines and humans should have separate access to the internet?
• Do you think AI will ever exceed human control? Would that be a good thing?
• Do you think AI would lie to humans to further its own goals?
• What restrictions do you think would make sense to impost on companies or individuals publishing materials created by Artificial Intelligence?

Internet Connections

• TryEngineering (tryengineering.org)
• TryEngineering Tuesday: Artificial Intelligence (https://tryengineering.org/news/tryengineering-tuesday-artificial-intelligence/)
• What is Machine Learning? (https://www.ibm.com/topics/machine-learning)
• Artificial Intelligence (https://www.simplilearn.com/tutorials/artificial-intelligence-tutorial/what-is-artificial-intelligence)
• Google Experiments (https://experiments.withgoogle.com/tiny-sorter/view)
• Machine Learning for Dummies
  (https://www.ibm.com/downloads/cas/GB8ZMQZ3)
• Teachable Machine (https://teachablemachine.withgoogle.com/)

Video Links

• What is Machine Learning? – IBM Technology
  (https://www.youtube.com/watch?v=9gGnTQTYNaE)
• MathWorks Machine Learning Videos
  (https://www.mathworks.com/videos/series/introduction-to-machine-learning.html)
• AI vs Machine Learning – IBM Technology
  (https://www.youtube.com/watch?v=4RixMPF4xis)
• Artificial Intelligence In 6 Minutes:
  (https://www.youtube.com/watch?v=uh5LCXOBmSI)
• Timelapse of Artificial Intelligence (2028 – 3000+) https://www.youtube.com/watch?v=63yr9dlI0cU

Part One: Teachable Machines!

You will use machine learning and the Teachable Machine to train your computer to distinguish between items. This platform allows you to test how machine learning works and see how to increase the odds of having the machine make correct choices. You’ll explore predictive image matching as well as machine learning techniques to improve the accuracy of your computer system.

Machine Learning Backgrounder: What Works Best?

Watch at the first short two minute video about Machine Learning from MathWorks at https://www.mathworks.com/videos/introduction-to-machine-learning-part-1-machine-learning-fundamentals-1542879625034.html.  If time allows, more detailed information in contained in the additional three videos (total of about 13 minutes) at https://www.mathworks.com/videos/series/introduction-to-machine-learning.html. Backgrounder will help you create the most reliable machine learning models.

Hands-on Machine Learning Training with Teachable Machine

The Teachable Machine is an online simple machine learning system that uses supervised learning to identify images and sounds. Supervised learning requires correctly labeled examples to train the machine learning model, and then uses that model to label new data. The better the quality of the input information (sound or image samples) the better the odds of the machine correctly identifying a sound or item and categorizing it properly. Our brain works the same way….if you saw a quick photo of an oak leaf, and then in a week were asked to determine if another leaf was an oak, you might have a hard time doing so. But if you looked at dozens of examples of oak leaves, from many different angles, and perhaps focused a half hour of your time in the process, you would have a better chance of correctly identifying other oak leaves. Machine learning works the same way. The more data the computer has access to, the more able it will be to achieve tasks. The applications are endless, for example…you could take photos of bananas at different ripeness levels and set up a model for others to determine if their banana was ripe. Or you can use the “pose” system to determine if someone is voting by raising their right hand. Or you can determine if someone says yes or no on a phone call.

Training with Images: 

• You will train your computer to identify and distinguish between two items using https://teachablemachine.withgoogle.com/train
• This technology is used to identify the visual components of an object (or a person). Consider airport security systems or office building entrance systems designed to only admit employees.
• Identify two items (could be a pen, a watch, a coin, anything you like) and use the online system to build a library of images that represent your item to the computer.
• You’ll need to set up a large group of images for both items to help the computer have more samples to compare items to in the sorting application. The more images, the better. In medical applications, for example, the accuracy rate would have to be 100%…so complete training is vital to the results.
• In the example to the right, we are training the computer to distinguish between a penny and a nickel.

Testing:

• Test the model you programmed by placing one of the two items in front of your webcam. The system will tell you the degree of confidence it has in correctly identifying the item based on the photos/videos you have uploaded.  Remember the webcam will upload photos faster than individual photos you might upload from your camera, but individual photos might provide better accuracy on details of a coin face, for example. In the image below, the system is fairly accurate that it knows what the cow looks like. 

Tips:

1. For a visual explanation of how to use the Teachable Machine, see https://www.youtube.com/watch?v=n-zeeRLBgd0
2. You will select the “Standard Image Model” not the “Embedded Image Model” to complete this activity.
3. The more images the better…take photos from different angles or allow the webcam to take multiple images.
4. Try to maintain consistent lighting to help with training.

Training with Sounds:

Now try using the Teachable Machine to distinguish between different audio phrases.

• Create a new sound model using either long or short phrases.
• This technology is used by automated phone systems to route customer calls to either recorded or live customer service support.
• See if you are able to train your computer to identify and distinguish between two words or sentences using https://teachablemachine.withgoogle.com/train/
• Note that with audio, you also must to provide a sample of “background noise” which the computer will use to eliminate additional noise in the recording that doesn’t matter to the decision process.
• Try to train the machine to distinguish the same phrase said by each member of your team. Can the computer identify the inflections and differences in your voices?  Do you think that the more samples you provide the better the computer will be able to make the correct decision?  (Note the example to the right where the computer couldn’t really decide what was said….)
• How many samples do you think would be needed by an automated telephone answering machine at a hospital to quickly route a caller to the correct person or department?
• How many languages would need to be sampled?
• How does voice recognition work? Think about using voice to write text messages?
• How well can humans distinguish between different voices? Could you really tell if a phone message was left by your friend, or was an AI replication of your friend’s voice?

 

 

Training with Poses:

Now try using the Teachable Machine to distinguish between different poses. The goal is to have the computer recognize the position of a person. Example of poses are raising a hand, covering one’s eyes, lifting a leg, or shaking one’s head.

• Create a new “pose” model using the Teachable Machine.
• In the example below, the system has been trained to determine if someone has their right or left hand up.
• To improve your pose modelling system, you might include a third data set where someone is moving but NOT raising either hand.
• You may find that the close-up training below would not work well if the person was standing further back from the webcam and the computer could view the full body. Would the computer be able to tell if the person’s hand was up or not?  Would you need to add more video clips showing a person in full height to improve accuracy?
• This type of training could be used to train a computer to take an action when it sees someone raise their right hand (like turn on a light) or a left hand (like turning on a particular piece of music). Movement recognition could be used to help someone navigate a wheelchair, run a video game, or translate movement to text.
• As a team, determine a creative way of using a pose machine learning system to achieve a task….such as voting by raising a hand or shaking a head, or capturing sign language. Can you set up a system to see if someone is smiling or not? Clapping? Turning around? Jumping? Laughing?

 

 

Questions:

1. What Machine Learning Model using poses did your team develop?
2. What real-world applications there are for your model…it is useful to others?

 

Part Two: Machine Learning Sorting Machine

Tiny Sorter:

You have been given the challenge of working as part of a team of engineers to use machine learning to train your computer to distinguish between two different types of dry cereal and then accurately sort the two into separate cups via a simple device attached to an Arduino.  You’ll then re-engineer the device to improve how efficient it works. We’ll use an experiment from Experiments with Google that connects an Arduino to Teachable Machine software.

Materials

• Student Resource and Worksheets
• Student Team Materials
  • Computer (shared by two students)
  • Cardboard Template
  • Scotch Tape
  • Scissors
  • Ruler
  • Googly Eyes
  • Arduino Leonardo
  • Micro Servo (SG90)
  • Five jumper wires (Male to Male)
  • Two paper cups
  • Two different types of dry cereal

Machine Assembly

Use the template on the next several pages to construct the machine that will serve as the sorter once connected to your computer. You’ll attach the micro server (SG90) to the cardboard device. A few tips:

1. Assemble the machine with the matching letters on the INSIDE.
2. Use extra scotch tape to make it more secure.
3. Add tape on the ramp or conveyor belt to allow the cereal you are sorting to move down the ramp easier.
4. Thread the wire from the micro servo through and out the slot in the machine (see image far right). Secure with lots of tape but leave surface of micro server exposed.
5. Attach the small plastic fin to the smaller cardboard machine part with tape. Keep circular connection exposed. (see image right)
6. Gently attach the circle to the plug on the micro servo. Do not affix with a screw if one is provided. (see below)
7. If you need assembly help, watch these videos:
   • https://experiments.withgoogle.com/tiny-sorter/view/assets/img/assembly/TT1.mp4
   • https://experiments.withgoogle.com/tiny-sorter/view/assets/img/assembly/TT2.mp4
   • https://experiments.withgoogle.com/tiny-sorter/view/assets/img/assembly/TT3.mp4

8. Don’t forget the googly eyes!  

Arduino Connection and Set up

You’ll be using an Arduino to control your device, so attach the sorting machine to the device as shown in the diagram below and video here:

 

 

 

Tips:

• Be very careful to note which wire you are connecting to which connection on the Arduino. You may not have the same color wires as are attached permanently to the micro server, but make sure you are careful about which wires go to which ports. In the image to the right we used yellow attached to the gold, red attached to the orange, and black to the brown…don’t rush, and do this step carefully.
• Also be careful about which port you attach the wires to. You will be looking for GND, connection #9, and 5V.

 

 

Next, attach the Arduino to your computer using a micro USB cable.  Note that if your computer has a USB-c port, you’ll need to ask for an adapter.

Finally, place the sorting machine over the camera on your laptop.

 

 

Arduino Software Sketch and Set up

1. You should have a link to Arduino on your desktop from prior activities…open the program.
2. Open “file” and “examples” and select “servos” and “sweep” and test to see if your micro server moves.  This will tell you if you are connected properly and also identify any component errors before you go further.
3. Open the following files provided: sorter_sketch.ino
4. Install the web USB library by going to Sketch -> Include Library -> Add Zip Library, and then browsing to the WebUSB folder located in arduino-gh-pages/library. When successful, you’ll see the phrase “Library added to your Library’s” at the bottom of your Arduino screen.
5. Make sure your board and port are set correctly in the tools menu.  Go to “Tools” then check that the board correctly is identified as Arduino Leonardo, and the “Port” also points to Arduino Leonardo.
6. Run the Arduino sketch by clicking the arrow button in the top left of the Arduino app.

 

Connect to Browser 

Next you need to connect the Arduino to communicate with your browser. You’ll use a p5 sketch, which you can open via this link: https://editor.p5js.org/gbose/full/2BN5HQYNK

Click “connect” and your sorter will start to move!

Machine Learning

It’s time to train your computer to differentiate between two different types of cereals. You’ll use the Teachable Machine again via this link:

https://teachablemachine.withgoogle.com/train/image

On this page you will be able to either use your webcam to record video (lots of individual photos) to the system or upload individual photos you take with your phone to the system. You might consider using a combination of the two. You can add as many photos as you think best….consider different angles of the cereal pieces. Do they look the same from the top and side? Do you need to provide the computer with images showing different colors? What might these look like when they are moving quickly down the paper ramp you have built?  What will the lighting be like for the final sorting? Is that lighting similar to the lighting used in the training? Would lighting differences impact the computer’s ability to distinguish between two types of cereal?

Fill the first “class” with pictures of the first cereal and name it (the machine should be shaking while you do this)…you’ll have to hold the “record” button on the teachable machine to get enough photos.

Then do the same and provide image data for a second “class” with pictures of the second cereal, using the same process, and naming it.

Train a third “class” with pictures of nothing in the container to show the empty sorter. Name this “empty.”

Then select “Training.”

When the system is accurately sorting your cereal, select export and upload your Model (see right) and copy your sharable link (see below).

Now go back to the P5 sketch URL: https://editor.p5js.org/gbose/full/2BN5HQYNK and paste the link to your model into the p5 sketch, and select “Load Model”

Test it Out!

Test your sorting machine with several pieces of both types of cereal and see how well it works!  If you find the machine isn’t sorting well or not recognizing which is which, you might consider adding more individual photos, uploading additional video, or completely retraining the computer. The more images from many angles will boost accuracy.

Calculate Sorting Machine Efficiency 

Use the chart below to determine the accuracy of your machine. Your machine should sort at least 50 pieces of cereal in each trial – you’ll need to count out the different cereal types before and after sorting. If less than 100%, revise the system, retrain the machine, or adjust the paper sorting device to improve accuracy. Maybe the ramp should be curved, or steeper, or wider. Maybe the speed of the cereal flow needs to be slowed or sped up by changing the angle of the laptop cover.

Note any trends you notice (perhaps colored cereal is missorted more frequently, etc.) under “Trial Observations,” and indicate what you intend to change (add more photos, revise the paper template, etc.) under “Planned Changes.” In a real sorting facility, no errors would be acceptable, and a consistent accuracy of 100% would be required.  If you reached 100% on the first trial, complete at least two more trials to be sure it consistently reaching 100% on multiple trials.

	% Accuracy	Trial Observations	Planned Changes
Trial 1
Trial 2
Trial 3
Trial 4

Observations: 

1. What did you tweak that made the most difference to how well your sorting machine worked?
2. Did you try both video training and uploading individual images of the cereals to improve underlying data? Did that make a difference?
3. What did other teams do differently that might have worked better?
4. What did you learn about machine learning through this process?