diff options
| -rw-r--r-- | docs/HARDWARE | 88 |
1 files changed, 88 insertions, 0 deletions
diff --git a/docs/HARDWARE b/docs/HARDWARE new file mode 100644 index 0000000..aaa7c59 --- /dev/null +++ b/docs/HARDWARE @@ -0,0 +1,88 @@ +BIOS +==== + The important options in BIOS: + - IOMMU (Intel VT-d) - Enable hardware translation between physcal and bus addresses + - No Snoop - Disables hardware cache coherency between DMA and CPU + - Max Payload (MMIO Size) - Maximal (useful) payload for PCIe protocol + - Above 4G Decoding - This seesm to allow bus addresses wider than 32-bit + - Memory performance - Frequency, Channel-interleaving, Hardware prefetcher affect memory performance + + +IOMMU +===== + - As many PCI-devices can address only 32-bit memory, for DMA operation some address + translation mechanism is required (also it helps with security limiting PCI devices + to only allowed address range). There are several methods to achieve this. + * Linux provides so called Bounce Buffers (or SWIOTLB). This is just a small memory + buffer in the lower 4 GB of memory. The DMA is actually performed into this buffer + and data is, then, copied to the appropriate location. One problem with SWIOTLB + is that it does not gurantee 4K aligned address when mapping memory pages (to + optimally use space). This is not properly supported neither by NWLDMA nor by IPEDMA. + * Alternatively hardware IOMMU can be used which will provide hardware address + translation between physical and bus addresses. To allow it, we need to + allow the technology in the BIOS and in the kernel. + + Intel VT-d or AMD-Vi (AMD IOMMU) virtualization technologies have to be enabled + + Intel is enabled with "intel_iommu=on" kernel parameter (alternative is to build kernel with CONFIG_INTEL_IOMMU_DEFAULT_ON) + + Checking: dmesg | grep -e IOMMU -e DMAR -e PCI-DMA + +DMA Cache Coherency +=================== + DMA API distinguishes two types of memory coherent and non-coherent. + - For the coherent memory, the hardware will care for cache consistency. This is often + achieved by snooping (No Snoop should be disabled in the BIOS). Alternatively, the same + effect can be achieved by using non-cached memory. There is architectures with 100% + cache coherent memory and others where only part of memory is kept cache coherent. + For such architectures the coherent memory can be allocated with + dma_alloc_coheretnt(...) / dma_alloc_attrs(...) + * However, the coherent memory could be slow (especially on large SMP systems). Also + minimal allocation unit may be restricted to page. Therefore, it is useful to group + consistent mapping into the groups. + + - On other hand, it is possible to allocate streaming DMA memory which are synchronized + using: + pci_dma_sync_single_for_device / pci_dma_sync_single_for_cpu + + - It may happen that all memory is coherent anyway and we do not need to call this 2 + functions. Currently, it seems not required on x86_64 which may indicate that snooping + is performed for all available memory. On other hand, may be only because nothing + was get cached luckely so far. + + +PCIe Payload +============ + - Kind of MTU for PCI protocol. Higher the value, the lower will be slow down due to + protocol headers while streaming large amount of data. The current values can be checked + with 'lspci -vv'. For each device, there is 2 values: + * MaxPayload under DevCap which indicates MaxPayload supported by the dvice + * MaxPayload under DevCtl indicates MaxPayload negotiated between device and chipset. + Negotiated MaxPayload is a minimal value among all the infrastructure between the device + chipset. Normally, it is limited by the MaxPaylod supported by the PCIe root port on + the chipset. Most systems currently restricted to 256 bytes. + + +Memory Performance +================== + - Memory performance is quite critical as we currently tripple the PCIe bandwidth: + DMA writes to memory, we read memory (it is not in cache), we write memory. + - The most important to enable Channel Interleaving (otherwise a single-channel copy + will be performed). On other hand, Rank Interleaving does not matter much. + - On some motherboards (Asrock X79 for instance), when the memory speed is set + manually, the interleaving is switched off in AUTO mode. So, it is safer to set + interleaving manually on. + - Hardware prefetching helps a little bit and should be turned on + - Faster memory frequency helps. As we are streaming I guess this is more important + compared even to slighly higher CAS & RAS latencies, but I have not checked. + - The memory bank conflicts sometimes may significant harm performance. Bank conflict + will happen if we read and write from/to different rows of the same bank (also there + could be conflict with DMA operation). I don't have a good idea how to prevent this + now. + - The most efficient memcpy performance depends on CPU generation. For latest models, + AVX seems to be most efficient. Filling all AVX registers before writting increases + performance. It also gives quite much of performance, if multiple pages copied in + parallel (still first we reading from multiple pages and then writting to multiple + pages, see ssebench). + - Usage of HugePages makes performance more stable. Using page-locked memory does not + help at all. + - This still will give about 10 - 15 GB/s at max. On multiprocessor systems about 5 GB/s, + because of performance penalties due to snooping. Therefore, copying with multiple + threads is preferable. |