USB is getting a facelift!
In the beginning, there was USB 1.1, with the "low speed" and "full speed" devices (at 1 Mbps and 12 Mbps, respectively). Then USB 2.0 came along with "high speed" devices that ran at 480 Mbps. Now the new USB 3.0 bus specification defines "SuperSpeed" devices that run at 5 Gbps (5,120 Mbps).
Now that the bus specification is public, I can finally talk about the code I've been developing at work. I've been writing a Linux driver for xHCI (the new USB 3.0 host controller), and changing the Linux kernel stack to support USB 3.0 devices. On November 17th, I got to demo my work at the world's first USB 3.0 "SuperSpeed" Developers Conference.
This is a demo showing a USB 3.0 Mass Storage Device (commonly called a USB drive, thumb drive, or flash drive) prototype running under Linux with an unmodified Mass Storage Device driver. My Linux xHCI driver is necessary to communicate with the USB 3.0 device through the xHCI host controller prototype. The FPGA prototype was provided by Fresco Logic, a company that sells host controller and device IP.
The demo showed speeds that were about 3.5 times faster than USB 2.0 high speed devices. I expect this demo to be even faster when the device and host controller are implemented in silicon.
Details about USB 3.0
USB 3.0 is 10 times faster than USB 2.0. Roughly speaking, it means that a file that takes 30 minutes to transfer over USB 2.0 could take 3 minutes to transfer under USB 3.0.
USB 3.0 also provides better power management, which translates to longer laptop battery life. USB 3.0 is backwards compatible. That means you can plug all your USB 2.0 devices into a USB 3.0 port, or plug your USB 3.0 device into a USB 2.0 port. The USB 3.0 device will work at USB 2.0 speeds in the latter case, but that means consumers don't have to upgrade their PC or laptop to use USB 3.0 devices at the slower speed.
FAQ
Q: When will there be USB 3.0 host controllers and devices?
A: Jeff Ravencraft, USB-IF President and Chairman, estimates that USB 3.0 devices will be shipped as early as mid-2009. See his SuperSpeed DevCon keynote slides, page 14.
Q: When will Windows have USB 3.0 support?
A: In the SuperSpeed Developers Conference keynote, Sriram Rajagopalan of Microsoft announced that they needed to rearchitect the USB stack, and they would have USB 3.0 support in "Windows 7+". Windows 7 is Microsoft's operating system version after Vista.
From a later Windows USB 3.0 session, Lars Giusti's slides say, "Early input from some partners indicates they would like us to consider supporting it on Windows Vista and newer Windows OS." Of course, that is only a statement of a request for support, not an official statement of support for USB 3.0 under Windows Vista. Nothing has been said of Windows XP.
Basically, Windows users have been promised official USB 3.0 support for Windows 7, not Vista or XP or older OSes. Some other USB vendors might ship unofficial Windows drivers for other Windows OSes, but that is the official word from Redmond as of now.
Q: When will Linux have USB 3.0 support?
For Linux to have basic USB 3.0 support, two things need to be added. First, we need to add support to the Linux USB stack to handle the new device speed and other changes mandated by the USB 3.0 bus specification. Second, we need to have a driver for the xHCI host controller. A host controller is the hardware that sits behind your USB port and talks to the USB devices you plug in.
Now that the bus specification is public, I can start pushing the patches for the USB core changes. They will need to be reviewed and possibly changed before they make it into the mainline kernel. Once the changes make it into the mainline kernel, they'll be picked up by Linux Operating System Vendors like RedHat, Novell, and Ubuntu.
The xHCI host controller driver is a little trickier. The xHCI specification is not public yet. It's currently available under NDA with Intel as a 0.9 draft specification. Since it's not a public specification, I'm forbidden to ship code that would reveal what's in the specification. That means the xHCI driver can't be sent out for review by the whole Linux community until the xHCI specification is public. The driver is much bigger than the USB core code changes, so I know it will go through several review iterations before it gets accepted into the mainline kernel.
The beauty of this open source process is that you can watch the development by following the Linux USB mailing list. I'll also update my blog with any announcements about Linux USB 3.0.
Q: Ok, so what does "basic" Linux USB 3.0 support mean?
A: Basic support means that some features might be lacking. We might not have awesome power management right off the bat, or we might be missing USB 3.0 support from some class drivers, or, heck, some versions of my xHCI driver might crash your system. My driver will be marked EXPERIMENTAL for a reason. ;)
The Linux philosophy is to ship early, and ship often. When code is shipped, more eyes can look at it and improve the code. Shipping partially functional code is better than waiting until the code is "perfect" and having the community report some fatal flaw or, worse, not understand why you're trying to get changes in.
Q: Will there be a Linux compliance program?
A: The USB-IF has run, and always will run, the official USB compliance program. Devices and host controllers that pass the compliance suite are allowed to use the official USB logos on their products. There are no plans for a separate Linux compliance program.
However, I'm willing to test new devices and host controllers on an unofficial basis to make sure they work properly under Linux. You can contact me at my Intel work address at Sarah.A.Sharp at linux.intel.com.
About the SuperSpeed DevCon Windows Demo
I can't really talk about my Linux demo without talking about the Windows Demo that was in the SuperSpeed Developers Conference keynote. The demo was created by the Intel team that works closely with the USB-IF. It was the same demo they used for IDF Taipei in August 2008. The demo used the same Fresco Logic USB 3.0 prototypes and nearly the same PC system that I used.
Their goal was to show the maximum speed possible from the host controller and USB 3.0 device. To do that, they ran a simple compliance test suite that allocated a giant DMA buffer and sent data as fast as possible to the USB 3.0 device. The device was programmed to use the USB 3.0 protocol, but it was basically a loop back device. Their demo showed speeds of 318 MBps.
The "wire speed" of USB 3.0 is 5Gbps. The 8b10b encoding and upper-level protocol overhead make USB 3.0 applications see less than that. The projected max application bandwidth for USB 3.0 devices is in the 400MBps range. In comparison, the wire speed of USB 2.0 is 60MBps, and applications see around 35MBps.
About the SuperSpeed DevCon Linux Demo
Fresco Logic, a company selling USB 3.0 host controller and device IP, provided hardware prototypes for an xHCI host controller and a USB 3.0 Mass Storage Device. For the demo, the host controller and the device was implemented on the same FPGA. I didn't have time to get a two-card solution working.
The Linux demo copied Arjan's 5-second boot video to the MSD, and played it using mplayer. The demo then read the 64KB mass storage device using dd with various block sizes, and displayed the host controller bandwidth measurements on a traffic graph. Visitors could see traffic spikes as the dd commands started and completed. They could also see the bandwidth increase as the block size increased to 16KB. In trial runs, block sizes larger than 16KB did not increase performance.
The script forced the system to drop all caches between runs (with
echo 3 > /proc/sys/vm/drop_caches). This made sure that the video would
actually be fetched from the device, instead of being cached. The dd command
also used the direct I/O flag to ensure parts of the disk would not be cached.
The timing measurements were taken with ktime_get(), which uses high res timers. The goal was to measure the best possible bandwidth the operating system would see. This is the same measurement that the Windows demo used. The Windows compliance driver allocated one giant DMA buffer. The Linux stack set up DMA scatter-gather lists dynamically, as real applications requested data.
The measurements were fairly simple. When an URB was enqueued, the host controller driver would take a timestamp with ktime_get() just before the URB's buffer was passed off to the hardware. When the host controller interrupted the system to indicate buffer completion, the host controller driver would take another timestamp. The delta time and the number of bytes transferred was then passed off to a generic HCD statistics reporting module that would return the delta time and bytes pairs to userspace. I got this reporting working two days before the demo.
My wonderful husband, Jamey, found an open source real-time graphing program called Trend. It looked perfect for the demo, except for the fact that it would only update when it was given data. Since the HCD stats file would block when there wasn't data (we didn't want to spend too many CPU cycles returning zero), Jamey wrote a threaded C program to sample the input from the HCD stats file.
The program takes the HCD stats file as an input, calculates the throughput (in MBps) for each pair read from the stats file, and passes off the max throughput to Trend for each sample. It would pass zeros off to Trend if the file I/O was blocked. This meant the Trend graph updated at the same rate, even if there was no USB traffic. The Trend graph also auto-updated its vertical scale as the throughput increased. This seemed to confuse some people, but I thought the "growing" effect was fun.
The Windows demo saw around 318 MBps, while the Linux demo typically showed 125 MBps. I saw as high as 233 MBps while formatting the disk. dd is not the best application to use for performance testing; I only used to whip up a simple demo.
Application layer measurements showed poor performance (around 2 MBps). I think two things added fixed latencies between the application layer and the host controller hardware. First, there was a massive amount of debugging output in the host controller driver and mass storage driver. Second, I had placed some msleep() calls in the USB MSD driver so that I could see the debugging output and trigger a PCI analyzer at the same time. I didn't have time to take those out before I ran my demo. I need to run more tests to disable debugging and profile the upper layer stack for other bottlenecks.
If you're interested in seeing the demo run on an EHCI host controller and a high speed USB 2.0 MSD, you can view the Google video here, or download the higher resolution ogg here.
| link | 40 comment(s)
Posted by voidlogic at Sat Dec 13 14:47:01 2008
Very nice work.
You mention the discrepancy between the 60 MB/s wirespeed and the 35 MB/s USB 2.0 applications typically see. Does that mean if the a systems USB hardware was beefy enough and its drivers were optimal the full 60 MB would be possible?
Posted by Sarah Sharp at Sat Dec 13 23:16:15 2008
voidlogic: The discrepancy between the wire speed and the actual speed is partially due to data encoding and protocol overhead. For example, USB uses 8b10b, which uses 10 bits to encode 8 bits of actual data. Then there's Cyclic Redundancy Checks (CRCs) and headers that have to be sent with the data. Plus 10% of the bandwidth has to be saved for control transfers and there's protocol overhead to ACK packets. All that encoding and protocol overhead means you'll never see 60MBps, even if you had perfect hardware and software.
Posted by Techtoad at Sun Dec 14 04:16:47 2008
Wow! USB 3.0 looks awesome! I think that the Windows 7 development for USB 3.0 will be good considering that Windows still control most of the IT market, however, if Linux can support USB 3.0 before the release of Windows 7, the IT market may begin to pick up more of an interest in Linux. 10x the current USB 2.0 speed would be very impressive, solid state memory devices are the way to go, I can see the original Hard Drives being pushed out of the market within the next 5 years - with these kinds of speeds and the increasing storage capability of solid state, we wont have a need for hard drives any more. I am looking forward to the development of USB 3.0! Keep up the good work! - Kirk - www.techtoad.co.uk
Posted by MBH at Sun Dec 14 04:46:57 2008
I salute you (and your husband) for your great work!
I was wondering about the CPU overhead with USB 3.0. There's no mention of any CPU overhead whether for 2.0 or 3.0. Would there be a difference between Windows & Linux in that area?
Thank you & keep up the good work :)
Posted by roshi at Sun Dec 14 04:53:42 2008
normal boot 5 sec?! that sounds really good. hummmm even with the massive capacities and cheap price of current plate based hard drives, I would change even without thinking. Boot times are a pain. I would like to get more details about that boot. Maybe I'm getting the wrong idea.
Posted by Sarah Sharp at Sun Dec 14 16:59:24 2008
roshi: Yes, Linux can boot in five seconds. It involved a bit of tweaking by Arjan Van de Ven and Auke Kok, who also work in my group at Intel. You can check out the articles on their work at LWN and O'Reilly.
Posted by paulmarko at Sun Dec 14 18:39:46 2008
USB video cards and monitors anyone? I know it doesn't match up exactly, but AGP 8x had up to 2GB/Sec! So USB 3.0 could have twice the throughput! (in theory)
That'd be GREAT for productivity! You might even be able to run 3 monitors off of one laptop! Sweet!
Posted by ntopics at Sun Dec 14 18:43:15 2008
I heard about USB 3 about a month ago.
Great to get a reminder of how fast its
going to be. Nice work on the adapter.
thanks from tony
Posted by Scott Glazer at Sun Dec 14 19:43:20 2008
paulmarko: The problem with that thinking is that, as per this article, USB 3.0's maximum throughput in their IDF demo was 318MB/s. You're also comparing 2GB/s with 5Gb/s. The difference is a factor of 8 (bytes vs. bits respectively), so 2GB = 16Gb/s, and 5Gbps doesn't even come close -- even with the protocol overhead. There are some USB 2.0 VGA adapters on the market, but don't expect to play 3D games on them.
Posted by Clint Brothers at Sun Dec 14 21:10:08 2008
Great job Sarah, I'm so glad a GNU/LINUX team is on top of this. It will be so nice to plug in a 500 GB SuperSpeed USB HDD into my Jaunty Jackalope box and back it up in 18 minutes instead of 3 hours. Keep up the great work!
Posted by Anonymous at Mon Dec 15 11:08:20 2008
I didnt understand stuff since I am into application development and not into system programming. But pretty impressive stuff and a well-written article.
Keep it up and keep such stuff coming.
Posted by sileNT at Mon Dec 15 13:44:18 2008
Finally, USB-sticks faster than HDD drives? Oh wait, looks like flash-media need to catch up with USB3 speed first.
Posted by Justin at Mon Dec 15 15:03:52 2008
This will open up possibilities to USB-powered gadgets! Think of all the things you can sync with using USB3.0!
Posted by eicc at Mon Dec 15 22:22:01 2008
well, this is very awesome, because exceeds by far the USB 2.0 and this makes me think that Linux is ready for the next devices under USB 3.0.
Posted by eivind at Wed Dec 17 15:03:48 2008
Thanks for a very interesting and also well written and detailed article, Sarah.
You write that the lower throughput for the demo with dd on GNU/Linux compared to the throughput in the Windows-demo was due to debug code and added msleep()s you didn't have time to remove.
Have you done any new benchmarking on GNU/Linux with this code stripped out? Do you care to share the results? ;-)
Keep up the good work. I'm looking forward to use USB 3.0.
Posted by Sarah Sharp at Fri Dec 19 01:06:26 2008
eivind: The lower throughput seen from userspace is explained by the debug code and msleep()s between the host controller driver and the application level. That doesn't explain the difference between the 125MBps seen in the Linux demo and the 318MBps seen in the Windows demo. Both of those numbers were at the measured in the host controller driver.
What might explain the ~200MBps difference is the fact that the Windows demo was allocating one contiguous buffer, while Linux was passing down a buffer of a similar size in several different pieces. The Windows was able to allocate its own memory, but the Linux demo had to deal with whatever buffer userspace handed it. The buffer would have been contiguous in virtual memory, but not contiguous in physical memory. The Linux host controller driver would have had to pass several DMA pointers for the hardware to fetch from, while the Windows demo would have passed only one DMA pointer.
I haven't done any new benchmarking yet. (Been too busy with other end-of-the-year stuff at work.) I will post again here on my blog when I run the demo on the bigger mass storage device. Taking in other people's suggestions, I'll show CPU utilization and test with a more standard disk benchmarking tool.
Posted by Akiba at Tue Dec 23 01:12:53 2008
Most individual hard drives max out at around 50-80 MBps. I think the WD Raptors are around 80 MBps. If you want to really exercise something like USB 3.0, you'll probably need to set up a 4-disk RAID 0 array. In my tests, using a 4-disk RAID 0 with an eSATA interface would get about 220 MBps which would seem to pair well with USB 3.0 at the real throughput numbers shown.
Posted by Allen at Sun Jan 4 11:15:03 2009
Great to see the hardware pushed but what will max out USB 30?
NAND gate technology is far too slow even with multiple data paths.
Hard drives can push high bandwidths but only for some very large files that are contigous under RAID-10 as Akiba pointed out.
HD TV codec is only pushing 1 gig/min.
For all my negativity I am looking forward to using both US 2.0 and newer 3.0 form factor as "things expand to overwhelm the space and time provided"
Regards,
Allen
"Most programs run in 8k so 640k will be more than enough" - Bill Gates, founder of MicroSoft.
Posted by harsha at Tue Feb 3 18:26:01 2009
Hi Sarah, first of all, Congratulations ! and kudos for your splendid work done.
However I am keen to know about the benchmarking criteria for this difference in speed between Windows and Linux. Why does this occur ? Any idea ?
Thanks for your attention in advance
Posted by USB3.0_Enthusiast at Wed Feb 25 22:15:09 2009
Hi Sarah,
Hope you are well.
I am interested in knowing about the Open source Linux xHCI driver support. I appreciate if you could provide relevant link / source to get the driver. Infact, I want to develop a test driver on the top of xHCD (xHCI) so that certain test applications can be written for Test Driver and send it to the External Device / Host too !
So please provide some hints / clues on xHCI driver. Thanks
Posted by Sarah Sharp at Wed Feb 25 23:39:15 2009
USB3.0_Enthusiast: The source code for the Linux xHCI driver is not available yet. Why do you want to build a test driver on top of it? You should probably discuss your software on the Linux USB mailing list: http://www.linux-usb.org/mailing.html
Posted by USB3.0_Enthusiast at Thu Feb 26 17:23:17 2009
Hi Sarah, This test driver of mine would intend to call main functions from the Standard xHCI so that certain custom-specific test applications could be well handled on the top of my test driver which serves the required purpose.
The main objective is to build up Mass Storage application on the USB Host PDK system and test / enumerate / execute against our company-specific USB 3.0 IP device to attain throughput performance of 150 - 250 Mbps. I am not sure if we could attain this speed, but need to fine-tune PHY transmit / Channel port allotments and transport layer for which xHCI would be the ultimate player.
I appreciate if you could throw some light about your Development / Test Environment setup at the Host-side platform as well as Device side.
Thanks in advance.
Posted by Sarah Sharp at Thu Feb 26 21:03:31 2009
USB3.0_Ethusiast: Your test application sounds awfully like the current Linux mass storage device driver. You should look into modifying it instead of writing a whole new driver. You could add specific sysfs interfaces to trigger the MSD driver to start a new test.
It would be really nice if you could open source a Linux driver (or test bench) for your device. Feel free to contact me at sarah.a.sharp at Linux dot Intel dot com and discuss this further.
Posted by Robin L. Ore at Thu May 14 20:19:12 2009
Sarah, I found you looking for progress in USB 3.0 bus speed which I understood to be the main problem? What do you think of the new XMOS XC1-G4 Processor that was just announced? http://www.xmos.com What about using a USB 3.O system such as this combined with yours using an eSATA solution? For what I am doing in HDTV I need full bandwidth MPEG4 with Xbox over utility at wires at 200Mbps with enough to drive our own VSLID optical projector. I am trying to understand the max size for the screen. Power is not an issue. Thank you for your eloquent expertise. Robin
Posted by Robin L. Ore at Thu May 14 22:46:38 2009
I returned to your site and found these two entries here underneath my name that I did not put there. ?
Posted by ZK@Web Marketing Blog at Sat May 23 20:27:38 2009
Sharp asserts that USB 3.0 support for Linux means having to adapt the Linux USB stack to the new device speed and develop a new driver for the xHCI host controller. She is beginning the kernel patches for the USB changes and hoping that Red Hat, Novell and Ubuntu and other distros will begin picking them up. The work on the xHCI host controller will be a "little trickier," she says, because it is currently under a non-disclosure agreement as a 0.9 draft specification that isn't yet ready for Linux community review.
Posted by paul at Thu Jul 2 03:35:34 2009
Thanks for the article, it's really well written. Can you just confirm it is compatible with windows XP. Thanks for your help.
Posted by Sarah Sharp at Thu Jul 2 09:43:40 2009
Paul: This article is about USB 3.0 support for Linux, not for Windows XP. Microsoft will have to write new drivers for Windows. I don't know if they will have Windows XP support for USB 3.0; a vendor who sells xHCI host controllers may have to provide their own driver for Windows XP.
Posted by Nathen at Mon Jul 13 07:55:35 2009
Nice article. I guess we'll have to wait a few years for USB 3.0 to become available on common devices. Looking forwards to the boost in speed when it happens.
Posted by Massa at Thu Jul 16 06:52:15 2009
Hi,
What about OTG support ? I expect that portable devices could act as hosts...
Posted by Konferens i Stockholm City at Mon Nov 9 00:09:50 2009
thanks for such information, we are waiting for USB 3.0 version. hope it comes very soon.
thanks
Posted by tisno at Mon Nov 9 10:33:42 2009
nice sharing about linux and usb 3.0, but i think this is just information, usb 3.0 will come in 2010 or more late.
tisno
Posted by effi at Tue Nov 10 11:35:33 2009
Hi Sarah, This test driver of mine would intend to call main functions from the Standard xHCI so that certain custom-specific test applications could be well handled on the top of my test driver which serves the required purpose.I really appreciate the current advancement in technology.
Posted by Konferens Uppsala at Tue Nov 10 11:40:49 2009
Hi Sarah, This test driver of mine would intend to call main functions from the Standard xHCI so that certain custom-specific test applications could be well handled.I really appreciate the current advancement in technology.
Posted by cashnetusa at Mon Nov 16 14:29:38 2009
Nice article. I guess we'll have to wait a few years for USB 3.0 to become available on common devices. Looking forwards to the boost in speed when it happens.
Posted by Chris at Thu Nov 26 02:20:00 2009
This will be great when it makes it's way into hd camcorders. The speed boost will make my life much easier. :-)
