summaryrefslogtreecommitdiffstats
path: root/NOTES
blob: 47b2d517331f972fcfc2ebc89acef782e8ae7586 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
Memory Addressing
=================
 There is 3 types of addresses: virtual, physical, and bus. For DMA a bus
 address is used. However, on x86 physical and  bus addresses are the same (on
 other architectures it is not guaranteed). Anyway, this assumption is still
 used by xdma driver, it uses phiscal address for DMA access. I have ported
 in the same way. Now, we need to provide additionaly bus-addresses in kmem
 abstraction and use it in NWL DMA implementation.

DMA Access Synchronization
==========================
 - At driver level, few types of buffers are supported:
    * SIMPLE - non-reusable buffers, the use infomation can be used for cleanup
    after crashed applications.
    * EXCLUSIVE - reusable buffers which can be mmaped by a single appliction
    only. There is two modes of these buffers:
	+ Buffers in a STANDARD mode are created for a single DMA operation and
	if such buffer is detected while trying to reuse, the last operation
	has failed and reset is needed.
	+ Buffers in a PERSISTENT mode are preserved between invocations of
	control application and cleaned up only after the PERSISTENT flag is 
	removed
    * SHARED - reusable buffers shared by multiple processes. Not really 
    needed at the moment.

    KMEM_FLAG_HW - indicates that buffer can be used by hardware, acually this
    means that DMA will be enabled afterwards. The driver is not able to check
    if it really was enable and therefore will block any attempt to release 
    buffer until KMEM_HW_FLAG is passed to kmem_free routine as well. The later
    should only called with KMEM_HW_FLAG after the DMA engine is stopped. Then,
    the driver can be realesd by kmem_free if ref count reaches 0.
    
    KMEM_FLAG_EXCLUSIVE - prevents multiple processes mmaping the buffer 
    simultaneously. This is used to prevent multiple processes use the same
    DMA engine at the same time. When passed to kmem_free, allows to clean
    buffers with lost clients even for shared buffers.
    
    KMEM_FLAG_REUSE - requires reuse of existing buffer. If reusable buffer is 
    found (non-reusable buffers, i.e. allocated without KMEM_FLAG_REUSE are
    ignored), it is returned instead of allocation. Three types of usage 
    counters are used. At moment of allocation, the HW reference is set if 
    neccessary. The usage counter is increased by kmem_alloc function and
    decreased by kmem_free. Finally, the reference is obtained at returned
    during mmap/munmap. So, on kmem_free, we do not clean
	a) buffers with reference count above zero or hardware reference set.
	REUSE flag should be supplied, overwise the error is returned
	b) PERSISTENT buffer. REUSE flash should be supplied, overwise the 
	error is returned
	c) non-exclusive buffers with usage counter above zero (For exclusive
	buffer the value of usage counter above zero just means that application
        have failed without cleaning buffers first. There is no easy way to 
        detect that for shared buffers, so it is left as manual operation in
        this case)
        d) any buffer if KMEM_FLAG_REUSE was provided to function
    During module unload, only buffers with references can prevent cleanup. In
    this case the only possiblity to free the driver is to call kmem_free 
    passing FORCE flags.
    
    KMEM_FLAG_PERSISTENT - if passed to allocation routine, changes mode of 
    buffer to PERSISTENT, if passed to free routine, vice-versa changes mode
    of buffer to NORMAL. Basically, if we call 'pci --dma-start' this flag
    should be passed to alloc and if we call 'pci --dma-stop' it should be
    passed to free. In other case, the flag should not be present.

    If application crashed, the munmap while be still called cleaning software
    references. However, the hardware reference will stay since it is not clear
    if hardware channel was closed or not. To lift hardware reference, the 
    application can be re-executed (or dma_stop called, for instance).
    * If there is no hardware reference, the buffers will be reused by next 
    call to application and for EXCLUSIVE buffer cleaned at the end. For SHARED
    buffers they will be cleaned during module cleanup only (no active 
    references).
    * The buffer will be reused by next call which can result in wrong behaviour
    if buffer left in incoherent stage. This should be handled on upper level.
    
 - At pcilib/kmem level synchronization of multiple buffers is performed
    * The HW reference and following modes should be consistent between member 
    parts: REUSABLE, PERSISTENT, EXCLUSIVE (only HW reference and PERSISTENT 
    mode should be checked, others are handled on dirver level)
    * It is fine if only part of buffers are reused and others are newly 
    allocated. However, on higher level this can be checked and resulting
    in failure.
    
    Treatment of inconsistencies:
     * Buffers are in PRESISTENT mode, but newly allocated, OK
     * Buffers are reused, but are not in PERSISTENT mode (for EXCLUSIVE buffers
     this means that application has crashed during the last execution), OK
     * Some of buffers are reused (not just REUSABLE, but actually reused), 
     others - not, OK until 
        a) either PERSISTENT flag is set or reused buffers are non-PERSISTENT
	b) either HW flag is set or reused buffers does not hold HW reference
     * PERSISTENT mode inconsistency, FAIL (even if we are going to set 
     PERSISTENT mode anyway)
     * HW reference inconsistency, FAIL (even if we are going to set 
     HW flag anyway)
     
    On allocation error at some of the buffer, call clean routine and
     * Preserve PERSISTENT mode and HW reference if buffers held them before
     unsuccessful kmem initialization. Until the last failed block, the blocks
     of kmem should be consistent. The HW/PERSISTENT flags should be removed
     if all reused blocks were in HW/PERSISTENT mode. The last block needs
     special treatment. The flags may be removed for the block if it was
     HW/PERSISTENT state (and others not).
     * Remove REUSE flag, we want to clean if allowed by current buffer status
     * EXCLUSIVE flag is not important for kmem_free routine.
    
 - At DMA level
    There is 4 components of DMA access:
    * DMA engine enabled/disabled
    * DMA engine IRQs enabled/disabled - always enabled at startup
    * Memory buffers
    * Ring start/stop pointers
    
    To prevent multiple processes accessing DMA engine in parallel, the first
    action is buffer initialization which will fail if buffers already used
	* Always with REUSE, EXCLUSIVE, and HW flags 
	* Optionally with PERSISTENT flag (if DMA_PERSISTENT flag is set)
    If another DMA app is running, the buffer allocation will fail (no dma_stop 
    is executed in this case) 

    Depending on PRESERVE flag, kmem_free will be called with REUSE flag 
    keeping buffer in memory (this is redundant since HW flag is enough) or HW
    flag indicating that DMA engine is stopped and buffer could be cleaned.
    PERSISTENT flag is defined by DMA_PERSISTENT flag passed to stop routine.
    
    PRESERVE flag is enforced if DMA_PERSISTENT is not passed to dma_stop
    routine and either it:
	a) Explicitely set by DMA_PERMANENT flag passed to dma_start 
	function 
	b) Implicitely set if DMA engine is already enabled during dma_start, 
	all buffers are reused, and are in persistent mode.
    If PRESERVE flag is on, the engine will not be stopped at the end of
    execution (and buffers will stay because of HW flag).
    
    If buffers are reused and are already in PERSISTENT mode, DMA engine was on 
    before dma_start (PRESERVE flag is ignored, because it can be enforced), 
    ring pointers are calculated from LAST_BD and states of ring elements.
    If previous application crashed (i.e. buffers may be corrupted). Two
    cases are possible:
    * If during the call buffers were in non-PERSISTENT mode, it can be 
    easily detected - buffers are reused, but are not in PERSISTENT mode 
    (or at least was not before we set them to). In this case we just 
    reinitialize all buffers.
    * If during the call buffers were in PERSISTENT mode, it is up to 
    user to check their consistency and restart DMA engine.]
    
    IRQs are enabled and disabled at each call
    
Register Access Synchronization
===============================
 We need to serialize access to the registers by the different running 
 applications and handle case when registers are accessed indirectly by
 writting PCI BARs (DMA implementations, for instance).

 - Module-assisted locking:
 * During initialization the locking context is created (which is basicaly
 a kmem_handle of type LOCK_PAGE. 
 * This locking context is passed to the kernel module along with lock type 
 (LOCK_BANK) and lock item (BANK ADDRESS). If lock context is already owns
 lock on the specified bank, just reference number is increased, otherwise
 we are trying to obtain new lock.
 * Kernel module just iterates over all registered lock pages and checks if
 any holds the specified lock. if not, the lock is obtained and registered
 in the our lock page.
 * This allows to share access between multiple threads of single application
 (by using the same lock page) or protect (by using own lock pages by each of
 the threads)
 * Either on application cleanup or if application crashed, the memory mapping
 of lock page is removed and, hence, locks are freed.
 
 - Multiple-ways of accessing registers
 Because of reference counting, we can successfully obtain locks multiple 
 times if necessary. The following locks are protecting register access:
  a) Global register_read/write lock bank before executing implementation
  b) DMA bank is locked by global DMA functions. So we can access the 
  registers using plain PCI bar read/write.
  c) Sequence of register operations can be protected with pcilib_lock_bank
  function
 Reading raw register space or PCI bank is not locked.
  * Ok. We can detect banks which will be affected by PCI read/write and 
  lock them. But shall we do it?
 
Register/DMA Configuration
==========================
 - XML description of registers
 - Formal XML-based (or non XML-based) language for DMA implementation. 
   a) Writting/Reading register values
   b) Wait until <register1>=<value> on <register2>=<value> report error
   c) ... ?