Kestrell

When the poorly-coded are leading the blind

Mon, 05 Apr 2021 15:03:39 GMT

Kes: Purrhaps this explains why 90% of all images get identified as cats.

From MIT Technology Review

April 1, 2021
The 10 most cited AI data sets are riddled with label errors, according to
a new study out of MIT,
https://arxiv.org/pdf/2103.14749.pdf
and it’s distorting our understanding of the field’s progress.

Data sets are the backbone of AI research, but some are more critical than others. There are a core set of them that researchers use to evaluate machine-learning models as a way to track how AI capabilities are advancing over time. One of the best-known is the canonical image-recognition data set ImageNet, which kicked off the modern AI revolution. There’s also MNIST, which compiles images of handwritten numbers between 0 and 9. Other data sets test models trained to recognize audio, text, and hand drawings.

In recent years, studies have found that these data sets can contain serious flaws. ImageNet, for example, contains
racist and sexist labels
https://excavating.ai/
as well as photos of people’s faces obtained without consent.
The latest study now looks at another problem: many of the labels are just flat-out wrong. A mushroom is labeled a spoon, a frog is labeled a cat, and a high note from Ariana Grande is labeled a whistle. The ImageNet test set has an estimated label error rate of 5.8%. Meanwhile, the test set for QuickDraw, a compilation of hand drawings, has an estimated error rate of 10.1%.
How was it measured? Each of the 10 data sets used for evaluating models has a corresponding data set used for training them. The researchers, MIT graduate students Curtis G. Northcutt and Anish Athalye and alum Jonas Mueller, used the training data sets to develop a machine-learning model and then used it to predict the labels in the testing data. If the model disagreed with the original label, the data point was flagged up for manual review. Five human reviewers on Amazon Mechanical Turk were asked to vote on which label—the model’s or the original—they thought was correct. If the majority of the human reviewers agreed with the model, the original label was tallied as an error and then corrected.

Does this matter? Yes. The researchers looked at 34 models whose performance had previously been measured against the ImageNet test set. Then they remeasured each model against the roughly 1,500 examples where the data labels were found to be wrong. They found that the models that didn’t perform so well on the original incorrect labels were some of the best performers after the labels were corrected. In particular, the simpler models seemed to fare better on the corrected data than the more complicated models that are used by tech giants like Google for image recognition and assumed to be the best in the field. In other words, we may have an inflated sense of how great these complicated models are because of flawed testing data.

Now what? Northcutt encourages the AI field to create cleaner data sets for evaluating models and tracking the field’s progress. He also recommends that
researchers improve their data hygiene when working with their own data. Otherwise, he says, “if you have a noisy data set and a bunch of models you’re
trying out, and you’re going to deploy them in the real world,” you could end up selecting the wrong model. To this end, he open-sourced

the code
https://github.com/cgnorthcutt/cleanlab
he used in his study for correcting label errors, which he says is already in use at a few major tech companies.

comments

Picture Smart improvements in Jaws 2021

Fri, 13 Nov 2020 12:41:55 GMT

Kes: I'm still amazed by this feature - the details are listed after the URL

What's New in Jaws, Zoom Text, and Fusion 2021
https://blog.freedomscientific.com/whats-new-in-jaws-zoomtext-and-fusion-2021/

Improvements to Picture Smart
Introduced in JAWS and Fusion 2019, the Picture Smart feature analyzes photos and displays a description in the Results Viewer, which can be read with JAWS. To use Picture Smart:

Press INSERT+SPACEBAR to activate layered commands.
Press P to activate the Picture Smart layer.
Press A, F, C, or B for a description, as described below:
Press A for a description of a photo acquired from a flatbed scanner or PEARL camera.
Press F for a description of an image file you selected in Windows Explorer.
Press C for a description of a control in focus. This can include a graphical button in a dialog box or other area of the screen.
Press B for a description of an image on the Windows Clipboard.
Several improvements to this feature are available in JAWS and Fusion 2021. These include:

Providing descriptions of images on web pages
Submitting images to multiple services for a more accurate analysis
Using Picture Smart with multiple languages
Learn more about Picture Smart in JAWS Help, or press INSERT+SPACEBAR, followed by P, then ? (question mark) for additional information.

comments

Microsoft and image recognition, 1Password, Jeopardy's online test now accessible

Thu, 22 Oct 2020 12:21:43 GMT

Kes: This first link discusses advances in Microsoft's image recognition technology, which is a pretty big deal for any visually impaired person using
Microsoft's free Seeing AI app
https://www.microsoft.com/en-us/ai/seeing-ai
or the Picture Smart feature in Jaws
https://blog.freedomscientific.com/picture-smart-in-jaws-independently-selecting-your-artwork/
which has been improved in the soon-to-be-released Jaws 2021
https://support.freedomscientific.com/Downloads/JAWS/JAWSPublicBeta

Microsoft Announces Breakthrough AI Image Captioning for Word, PowerPoint, Outlook
https://blogs.microsoft.com/ai/azure-image-captioning/

Web Friendly Help discusses new features in NVDA
https://webfriendlyhelp.com/new-features-in-nvda-2020-3/

Google Search Tips: Always Find What You're Looking For by Online Tech Tips
https://www.online-tech-tips.com/google-softwaretips/8-google-search-tips-always-find-what-youre-looking-for/

Using Zoom with a Screen Reader on the Eyes on Success podcast
The latest episode features Heather Thomas, author of _Getting Started with Zoom Meetings: A guide for Jaws, NVDA, and iPhone VoiceOver users_.
http://eyesonsuccess.net

1Password: Mosen at Large provides a review and demonstration of this password manager
https://mosenatlarge.pinecast.co/episode/a69030679a8d434b/review-and-demonstration-of-the-1password-password-manager

Facebook Mobile with Chrome and Edge: the mbasic interface is actually the older m.facebook.com interface
https://mbasic.facebook.com/

Jeopardy Makes Online Test Accessible to the Blind
https://www.nfb.org/about-us/press-room/jeopardy-makes-online-test-accessible-blind

RNIB Creates Accessible Pregnancy Test Prototype to Raise Awareness of Accessible Design
In 2020 there is still no fully accessible pregnancy test, meaning that blind and partially sighted women must ask for help to read their tests, and are therefore never the first to know what is happening to their own bodies. The Royal National Institute of Blind People (RNIB) has unveiled the first accessible pregnancy test prototype that allows women with sight loss to know their results privately for the first time. The groundbreaking test allows the user to feel their results, producing raised nodules to indicate a positive result. Learn more at:.
https://www.dexigner.com/news/33351

A Day in The Connected Digital Life on Tek Talk will discuss Apple products.
GMT Tuesday, October 27, 2020 at 00:00
https://zoom.us/j/839935813

Most of these links are culled from the Top Tech Tidbits weekly newsletter: you can view the entire newsletter or subscribe at
https://toptechtidbits.com/

comments