00001 /* ---------- 00002 * veil_mainpage.c 00003 * 00004 * Doxygen documentation root for Veil 00005 * 00006 * Copyright (c) 2005 - 2011 Marc Munro 00007 * Author: Marc Munro 00008 * License: BSD 00009 * 00010 */ 00011 00012 00013 /*! \mainpage Veil 00014 \version 9.1.0 (Stable)) 00015 \section license License 00016 BSD 00017 \section intro_sec Introduction 00018 00019 Veil is a data security add-on for Postgres. It provides an API 00020 allowing you to control access to data at the row, or even column, 00021 level. Different users will be able to run the same query and see 00022 different results. Other database vendors describe this as a Virtual 00023 Private Database. 00024 00025 \section Why Why do I need this? 00026 If you have a database-backed application that stores sensitive data, 00027 you will be taking at least some steps to protect that data. Veil 00028 provides a way of protecting your data with a security mechanism 00029 within the database itself. No matter how you access the database, 00030 whether you are a legitimate user or not, you cannot by-pass Veil 00031 without superuser privileges. 00032 00033 \subsection Advantages Veil Advantages 00034 By placing security mechanisms within the database itself we get a 00035 number of advantages: 00036 - Ubiquity. Security is always present, no matter what application or 00037 tool is used to connect to the database. If your application is 00038 compromised, your data is still protected by Veil. If an intruder gets 00039 past your outer defences and gains access to psql, your data is still 00040 protected. 00041 - Single Security Policy and Implementation. If you have N applications 00042 to secure, you have to implement your security policy N times. With 00043 Veil, all applications may be protected by a single implementation. 00044 - Strength in Depth. For the truly security conscious, Veil provides 00045 yet another level of security. If you want strength in depth, with 00046 layers and layers of security like an onion, Veil gives you that extra 00047 layer. 00048 - Performance. Veil is designed to be both flexible and efficient. 00049 With a good implementation it is possible to build access controls with 00050 a very low overhead, typically much lower than building the equivalent 00051 security in each application. 00052 - Cooperation. The Veil security model is designed to cooperate with your 00053 applications. Although Veil is primarily concerned with data access 00054 controls, it can also be used to provide function-level privileges. If 00055 your application has a sensitive function X, it can query the database, 00056 through Veil functions, to ask the question, "Does the current user have 00057 execute_X privilege?". Also, that privilege can be managed in exactly 00058 the same way as any other privilege. 00059 - Flexibility. Veil is a set of tools rather than a product. How you 00060 use it is up to you. 00061 00062 \subsection Limitations Veil Limitations 00063 Veil can restrict the data returned by queries but cannot prevent the 00064 query engine itself from seeing restricted data. In particular, 00065 functions executed during evaluation of the where clause of a query may 00066 be able to see data that Veil is supposed to restrict access to. 00067 00068 As an example let's assume that we have a secured view, users, that 00069 allows a user to see only their own record. When Alice queries the 00070 view, she will see this: 00071 00072 \verbatim 00073 select * from users; 00074 00075 userid | username 00076 ----------+----------- 00077 12345 | Alice 00078 \endverbatim 00079 00080 Alice should not be able to see any details for Bob or even, strictly 00081 speaking, tell whether there is an entry for Bob. This query though: 00082 00083 \verbatim 00084 select * from users 00085 where 0 = 9 / (case username when 'Bob' then 0 else 1 end); 00086 \endverbatim 00087 00088 will raise an exception if the where clause encounters the username 00089 'Bob'. So Alice can potentially craft queries that will enable her to 00090 discover whether specific names appear in the database. 00091 00092 This is not something that Veil is intended to, or is able to, prevent. 00093 Changes to the underlying query engine to attempt to plug such holes 00094 have been proposed, but they all have their limitations and are likely 00095 to lead to degraded performance. 00096 00097 A more serious problem occurs if a user is able to create user defined 00098 functions as these can easily provide covert channels for leaking data. 00099 Consider this query: 00100 00101 \verbatim 00102 select * from users where leak_this(username); 00103 \endverbatim 00104 00105 Although the query will only return what the secured view allows it to, 00106 the where clause, if the function is deemed inexpensive enough, will see 00107 every row in the table and so will be able to leak supposedly protected 00108 data. This type of exploit can be protected against easily enough by 00109 preventing users from defining their own functions, however there are 00110 postgres builtins that can be potentially be exploited in the same way. 00111 00112 \subsection GoodNews The Good News 00113 00114 The news is not all bad. Although Veil can be circumvented, as shown 00115 above, a database protected by Veil is a lot more secure than one which 00116 is not. Veil can provide extra security over and above that provided by 00117 your application, and in combination with a well designed application 00118 can provide security that is more than adequate for most commercial 00119 purposes. 00120 00121 \section the-rest Veil Documentation 00122 - \subpage overview-page 00123 - \subpage API-page 00124 - \subpage Building 00125 - \subpage Demo 00126 - \subpage Management 00127 - \subpage Esoteria 00128 - \subpage install 00129 - \subpage History 00130 - \subpage Feedback 00131 - \subpage Performance 00132 - \subpage Credits 00133 00134 Next: \ref overview-page 00135 00136 */ 00137 /*! \page overview-page Overview: a quick introduction to Veil 00138 00139 \section Overview-section Introduction 00140 The section introduces a number of key concepts, and shows the basic 00141 components of a Veil-protected system: 00142 - \ref over-views 00143 - \ref over-connections 00144 - \ref over-privs 00145 - \ref over-contexts 00146 - \ref over-funcs2 00147 - \ref over-roles 00148 00149 \subsection over-views Secured Views and Access Functions 00150 Access controls are implemented using secured views and instead-of triggers. 00151 Users connect to an account that has access only to the secured views. 00152 For a table defined thus: 00153 \verbatim 00154 create table persons ( 00155 person_id integer not null, 00156 person_name varchar(80) not null 00157 ); 00158 \endverbatim 00159 The secured view would be defined something like this: 00160 \verbatim 00161 create view persons( 00162 person_id, person_name) as 00163 select person_id, person_name 00164 from persons 00165 where i_have_personal_priv(10013, person_id); 00166 \endverbatim 00167 00168 A query performed on the view will return rows only for those persons 00169 where the current user has privilege 10013 00170 (<code>SELECT_PERSONS</code>). We call the function 00171 <code>i_have_personal_priv()</code>, an access function. Such 00172 functions are user-defined, and are used to determine whether the 00173 connected user has a specific privilege in any of a number of security 00174 contexts (see \ref over-contexts). The example above is 00175 taken from the Veil demo application (\ref demo-sec) and 00176 checks for privilege in the global and personal contexts. 00177 00178 \subsection over-connections The Connected User and Connection Functions 00179 To determine a user's privileges, we have to know who that user is. 00180 At the start of each database session the user must be identified, and 00181 their privileges must be determined. This is done by calling a 00182 connection function, eg: 00183 \verbatim 00184 select connect_person('Wilma', 'AuthenticationTokenForWilma'); 00185 \endverbatim 00186 The connection function performs authentication, and stores the user's 00187 access privileges in Veil state variables. These variables are then 00188 interrogated by the access functions used in the secured views. 00189 00190 Prior to connection, or in the event of the connection failing, the 00191 session will have no privileges and will probably be unable to see any 00192 data. Like access functions, connection functions are user-defined and 00193 may be written in any language supported by PostgreSQL. 00194 00195 \subsection over-privs Privileges 00196 Veil-based systems define access rights in terms of privileges. A 00197 privilege is a named thing with a numerical value (actually, the name 00198 is kind of optional). 00199 00200 An example will probably help. Here is a definition of a privileges 00201 table and a subset of its data: 00202 \verbatim 00203 create table privileges ( 00204 privilege_id integer not null, 00205 privilege_name varchar(80) not null 00206 ); 00207 00208 copy privileges (privilege_id, privilege_name) from stdin; 00209 10001 select_privileges 00210 10002 insert_privileges 00211 10003 update_privileges 00212 10004 delete_privileges 00213 . . . 00214 10013 select_persons 00215 10014 insert_persons 00216 10015 update_persons 00217 10016 delete_persons 00218 10017 select_projects 00219 10018 insert_projects 00220 10019 update_projects 00221 10020 delete_projects 00222 . . . 00223 10100 can_connect 00224 \. 00225 00226 \endverbatim 00227 Each privilege describes something that a user can do. It is up to the 00228 access and connection functions to make use of these privileges; the 00229 name of the privilege is only a clue to its intended usage. In the 00230 example we might expect that a user that has not been given the 00231 <code>can_connect</code> privilege would not be able to authenticate 00232 using a connection function but this is entirely dependent on the 00233 implementation. 00234 00235 \subsection over-contexts Security Contexts 00236 00237 Users may be assigned privileges in a number of different ways. They 00238 may be assigned directly, indirectly through various relationships, or 00239 may be inferred by some means. To aid in the discussion and design of a 00240 Veil-based security model we introduce the concept of security 00241 contexts, and we say that a user has a given set of privileges in a 00242 given context. There are three types of security context: 00243 00244 - Global Context. This refers to privileges that a user has been given 00245 globally. If a user has <code>select_persons</code> privilege in the 00246 global context, they will be able to select every record in the 00247 persons table. Privileges in global context are exactly like 00248 database-level privileges: there is no row-level element to them. 00249 00250 - Personal Context. This context provides privileges on data that you 00251 may be said to own. If you have <code>select_persons</code> 00252 privilege in only the personal context, you will only be able to 00253 select your own persons record. Assignment of privileges in the 00254 personal context is often defined implicitly or globally, for all 00255 users, rather than granted explicitly to each user. It is likely 00256 that everyone should have the same level of access to their own data 00257 so it makes little sense to have to explicitly assign the privileges 00258 for each individual user. 00259 00260 - Relational Contexts. These are the key to most row-level access 00261 controls. Privileges assigned in a relational context are assigned 00262 through relationships between the connected user and the data to be 00263 accessed. Examples of relational contexts include: assignments to 00264 projects, in which a user will gain access to project data only if 00265 they have been assigned to the project; and the management hierarchy 00266 within a business, in which a manager may have specific access to 00267 data about a staff member. Note that determining a user's access 00268 rights in a relational context may require extra queries to be 00269 performed for each function call. Your design should aim to minimise 00270 this. Some applications may require several distinct relational 00271 contexts. 00272 00273 \subsection over-funcs2 Access Functions and Security Contexts 00274 Each access function will operate on privileges for a specific set of 00275 contexts. For some tables, access will only be through global context. 00276 For others, it may be through global and personal as well as a number of 00277 different relational contexts. Here, from the demo application, are a 00278 number of view definitions, each using a different access function that 00279 checks different contexts. 00280 \verbatim 00281 create view privileges( 00282 privilege_id, 00283 privilege_name) as 00284 select privilege_id, 00285 privilege_name 00286 from privileges 00287 where i_have_global_priv(10001); 00288 00289 . . . 00290 00291 create view persons( 00292 person_id, 00293 person_name) as 00294 select person_id, 00295 person_name 00296 from persons 00297 where i_have_personal_priv(10013, person_id); 00298 00299 . . . 00300 00301 create view projects( 00302 project_id, 00303 project_name) as 00304 select project_id, 00305 project_name 00306 from projects 00307 where i_have_project_priv(10017, project_id); 00308 00309 . . . 00310 00311 create view assignments ( 00312 project_id, 00313 person_id, 00314 role_id) as 00315 select project_id, 00316 person_id, 00317 role_id 00318 from assignments 00319 where i_have_proj_or_pers_priv(10025, project_id, person_id); 00320 \endverbatim 00321 00322 In the <code>privileges</code> view, we only check for privilege in the 00323 global context. This is a look-up view, and should be visible to all 00324 authenticated users. 00325 00326 The <code>persons</code> view checks for privilege in both the global 00327 and personal contexts. It takes an extra parameter identifying the 00328 person who owns the record. If that person is the same as the connected 00329 user, then privileges in the personal context may be checked. If not, 00330 only the global context applies. 00331 00332 The <code>projects</code> view checks global and project contexts. The 00333 project context is a relational context. In the demo application, a 00334 user gains privileges in the project context through assignments. An 00335 assignment is a relationship between a person and a project. Each 00336 assignment record has a role. This role describes the set of privileges 00337 the assignee (person) has within the project context. 00338 00339 The <code>assignments</code> view checks all three contexts (global, 00340 personal and project). An assignment contains data about a person and a 00341 project so privileges may be acquired in either of the relational 00342 contexts, or globally. 00343 00344 \subsection over-roles Grouping Privileges by Roles 00345 Privileges operate at a very low-level. In a database of 100 tables, 00346 there are likely to be 500 to 1,000 privileges in use. Managing 00347 users access at the privilege level is, at best, tedious. Instead, we 00348 tend to group privileges into roles, and assign only roles to individual 00349 users. Roles act as function-level collections of privileges. For 00350 example, the role <code>project-readonly</code> might contain all of the 00351 <code>select_xxx</code> privileges required to read all project data. 00352 00353 A further refinement allows roles to be collections of sub-roles. 00354 Defining suitable roles for a system is left as an exercise for the 00355 reader. 00356 00357 Next: \ref API-page 00358 00359 */ 00360 /*! \page API-page The Veil API 00361 \section API-sec The Veil API 00362 This section describes the Veil API. It consists of the following 00363 sections 00364 00365 - \ref API-intro 00366 - \subpage API-variables 00367 - \subpage API-simple 00368 - \subpage API-bitmaps 00369 - \subpage API-bitmap-arrays 00370 - \subpage API-bitmap-hashes 00371 - \subpage API-int-arrays 00372 - \subpage API-serialisation 00373 - \subpage API-control 00374 00375 Note that all veil objects are placed in the veil schema. 00376 00377 \section API-intro Veil API Overview 00378 Veil is an API that simply provides a set of state variable types, and 00379 operations on those variable types, which are optimised for privilege 00380 examination and manipulation. 00381 00382 The fundamental data type is the bitmap. Bitmaps are used to 00383 efficiently record and test sets of privileges. Bitmaps may be combined 00384 into bitmap arrays, which are contiguous groups of bitmaps indexed by 00385 integer, and bitmap hashes which are non-contiguous and may be indexed 00386 by text strings. 00387 00388 In addition to the bitmap-based types, there are a small number of 00389 support types that just help things along. If you think you have a case 00390 for defining a new type, please 00391 \ref Feedback "contact" 00392 the author. 00393 00394 Next: \ref API-variables 00395 */ 00396 /*! \page API-variables Variables 00397 Veil variables exist to record session and system state. They retain 00398 their values across transactions. Variables may be defined as either 00399 session variables or shared variables. 00400 00401 All variables are referenced by name; the name of the variable is 00402 passed as a text string to Veil functions. 00403 00404 Session variables are private to the connected session. They are 00405 created when first referenced and, once defined, their type is set for 00406 the life of the session. 00407 00408 Shared variables are global across all sessions. Once a shared variable 00409 is defined, all sessions will have access to it. Shared variables are 00410 defined in two steps. First, the variable is defined as shared, and 00411 then it is initialised and accessed in the same way as for session 00412 variables. Note that shared variables should only be created within 00413 \ref API-control-registered-init or \ref API-control-init. 00414 00415 Note that bitmap refs and bitmap hashes may not be stored in shared 00416 variables. 00417 00418 The following types of variable are supported by Veil, and are described 00419 in subsequent sections: 00420 - integers 00421 - ranges 00422 - bitmaps 00423 - bitmap refs 00424 - bitmap arrays 00425 - bitmap hashes 00426 - integer arrays 00427 00428 The following functions comprise the Veil variables API: 00429 00430 - <code>\ref API-variables-share</code> 00431 - <code>\ref API-variables-var</code> 00432 00433 Note again that session variables are created on usage. Their is no 00434 specific function for creating a variable in the variables API. For an 00435 example of a function to create a variable see \ref API-bitmap-init. 00436 00437 \section API-variables-share share(name text) 00438 \verbatim 00439 function veil.share(name text) returns bool 00440 \endverbatim 00441 00442 Implemented by C function veil_share(), this is used to define a 00443 specific variable as being shared. A shared variable is accessible to 00444 all sessions and exists to reduce the need for multiple copies of 00445 identical data. For instance in the Veil demo, role_privileges are 00446 recorded in a shared variable as they will be identical for all 00447 sessions, and to create a copy for each session would be an unnecessary 00448 overhead. This function should only be called from 00449 \ref API-control-registered-init or \ref API-control-init. 00450 00451 \section API-variables-var veil_variables() 00452 \verbatim 00453 function veil.veil_variables() returns setof veil_variable_t 00454 \endverbatim 00455 00456 This function, implemented by C function veil_variables(), returns a 00457 description for each variable known to the session. It provides the 00458 name, the type, and whether the variable is shared. It is primarily 00459 intended for interactive use when developing and debugging Veil-based 00460 systems. 00461 00462 Next: \ref API-simple 00463 */ 00464 /*! \page API-simple Basic Types: Integers and Ranges 00465 00466 Veil's basic types are those that do not contain repeating groups 00467 (arrays, hashes, etc). 00468 00469 Ranges, implemented by the type \ref veil_range_t, 00470 consist of a pair of values and are generally used to initialise the 00471 bounds of array and bitmap types. 00472 00473 \anchor veil_range_t \ref veil_range_t is defined as: 00474 00475 \verbatim 00476 create type veil.veil_range_t as ( 00477 min int4, 00478 max int4 00479 ); 00480 \endverbatim 00481 00482 Ranges may not contain nulls. 00483 00484 The int4 type is used to record a simple nullable integer. This is 00485 typically used to record the id of the connected user in a session. 00486 00487 The following functions comprise the Veil basic types API: 00488 00489 - <code>\ref API-basic-init-range</code> 00490 - <code>\ref API-basic-range</code> 00491 - <code>\ref API-basic-int4-set</code> 00492 - <code>\ref API-basic-int4-get</code> 00493 00494 \section API-basic-init-range init_range(name text, min int4, max int4) 00495 \verbatim 00496 function veil.init_range(name text, min int4, max int4) returns int4 00497 \endverbatim 00498 00499 This, implemented by veil_init_range() defines a range, and returns the 00500 extent of that range. 00501 00502 \section API-basic-range range(name text) 00503 \verbatim 00504 function veil.range(name text) returns veil.range_t 00505 \endverbatim 00506 00507 This, implemented by C function veil_range() returns the contents 00508 of a range. It is intended primarily for interactive use. 00509 00510 \section API-basic-int4-set int4_set(name text, value int4) 00511 \verbatim 00512 function veil.int4_set(name text, value int4) returns int4 00513 \endverbatim 00514 00515 Sets an int4 variable to a value, returning that same value. It is 00516 implemented by C function veil_int4_set(). 00517 00518 \section API-basic-int4-get int4_get(name text) 00519 \verbatim 00520 function veil.int4_get(name text) returns int4 00521 \endverbatim 00522 00523 Returns the value of the int4 variable given by name. Implemented by C 00524 function veil_int4_get(). 00525 00526 00527 Next: \ref API-bitmaps 00528 */ 00529 /*! \page API-bitmaps Bitmaps and Bitmap Refs 00530 Bitmaps are used to implement bounded sets. Each bit in the bitmap may 00531 be on or off representing presence or absence of a value in the set. 00532 Typically bitmaps are used to record sets of privileges. 00533 00534 A bitmap ref is a variable that may temporarily reference another 00535 bitmap. These are useful for manipulating specific bitmaps within 00536 bitmap arrays or bitmap hashes. All bitmap operations except for \ref 00537 API-bitmap-init may take the name of a bitmap ref instead of a bitmap. 00538 00539 Bitmap refs may not be shared, and the reference is only accessible 00540 within the transaction that created it. These restrictions exist to 00541 eliminate the possibility of references to deleted objects or to objects 00542 from other sessions. 00543 00544 The following functions comprise the Veil bitmaps API: 00545 00546 - <code>\ref API-bitmap-init</code> 00547 - <code>\ref API-bitmap-clear</code> 00548 - <code>\ref API-bitmap-setbit</code> 00549 - <code>\ref API-bitmap-clearbit</code> 00550 - <code>\ref API-bitmap-testbit</code> 00551 - <code>\ref API-bitmap-union</code> 00552 - <code>\ref API-bitmap-intersect</code> 00553 - <code>\ref API-bitmap-bits</code> 00554 - <code>\ref API-bitmap-range</code> 00555 00556 \section API-bitmap-init init_bitmap(bitmap_name text, range_name text) 00557 \verbatim 00558 function veil.init_bitmap(bitmap_name text, range_name text) returns bool 00559 \endverbatim 00560 This is used to create or resize a bitmap. The first parameter provides 00561 the name of the bitmap, the second is the name of a range variable that 00562 will govern the size of the bitmap. It is implemented by C function 00563 veil_init_bitmap(). 00564 00565 \section API-bitmap-clear clear_bitmap(bitmap_name text) 00566 \verbatim 00567 function veil.clear_bitmap(bitmap_name text) returns bool 00568 \endverbatim 00569 This is used to clear (set to zero) all bits in the bitmap. It is 00570 implemented by C function veil_clear_bitmap(). 00571 00572 \section API-bitmap-setbit bitmap_setbit(bitmap_name text, bit_number int4) 00573 \verbatim 00574 function veil.bitmap_setbit(bitmap_name text, bit_number int4) returns bool 00575 \endverbatim 00576 This is used to set a specified bit, given by bit_number in the bitmap 00577 identified by bitmap_name. It is implemented by C function 00578 veil_bitmap_setbit(). 00579 00580 \section API-bitmap-clearbit bitmap_clearbit(bitmap_name text, bit_number int4) 00581 \verbatim 00582 function veil.bitmap_clearbit(bitmap_name text, bit_number int4) returns bool 00583 \endverbatim 00584 This is used to clear (set to zero) a specified bit in a bitmap. It is 00585 implemented by C function veil_bitmap_clearbit(). 00586 00587 \section API-bitmap-testbit bitmap_testbit(bitmap_name text, bit_number int4) 00588 \verbatim 00589 function veil.bitmap_testbit(bitmap_name text, bit_number int4) returns bool 00590 \endverbatim 00591 This is used to test a specified bit in a bitmap. It returns true if 00592 the bit is set, false otherwise. It is implemented by C function 00593 veil_bitmap_testbit(). 00594 00595 \section API-bitmap-union bitmap_union(result_name text, bm2_name text) 00596 \verbatim 00597 function veil.bitmap_union(result_name text, bm2_name text) returns bool 00598 \endverbatim 00599 Form the union of two bitmaps with the result going into the first. 00600 Implemented by C function veil_bitmap_union(). 00601 00602 \section API-bitmap-intersect bitmap_intersect(result_name text, bm2_name text) 00603 \verbatim 00604 function veil.bitmap_intersect(result_name text, bm2_name text) returns bool 00605 \endverbatim 00606 Form the intersection of two bitmaps with the result going into the 00607 first. Implemented by C function veil_bitmap_intersect(). 00608 00609 \section API-bitmap-bits bitmap_bits(bitmap_name text) 00610 \verbatim 00611 function veil.bitmap_bits(bitmap_name text) returns setof int4 00612 \endverbatim 00613 This is used to list all bits set within a bitmap. It is primarily for 00614 interactive use during development and debugging of Veil-based systems. 00615 It is implemented by C function veil_bitmap_bits(). 00616 00617 \section API-bitmap-range bitmap_range(bitmap_name text) 00618 \verbatim 00619 function veil.bitmap_range(bitmap_name text) returns veil.range_t 00620 \endverbatim 00621 This returns the range, as a \ref veil_range_t, of a 00622 bitmap. It is primarily intended for interactive use. It is 00623 implemented by C function veil_bitmap_range(). 00624 00625 Next: \ref API-bitmap-arrays 00626 */ 00627 /*! \page API-bitmap-arrays Bitmap Arrays 00628 A bitmap array is an array of identically-ranged bitmaps, indexed 00629 by an integer value. They are initialised using two ranges, one for the 00630 range of each bitmap, and one providing the range of indices for the 00631 array. 00632 00633 Typically bitmap arrays are used for collections of privileges, where 00634 each element of the collection is indexed by something like a role_id. 00635 00636 The following functions comprise the Veil bitmap arrays API: 00637 00638 - <code>\ref API-bmarray-init</code> 00639 - <code>\ref API-bmarray-clear</code> 00640 - <code>\ref API-bmarray-bmap</code> 00641 - <code>\ref API-bmarray-testbit</code> 00642 - <code>\ref API-bmarray-setbit</code> 00643 - <code>\ref API-bmarray-clearbit</code> 00644 - <code>\ref API-bmarray-union</code> 00645 - <code>\ref API-bmarray-intersect</code> 00646 - <code>\ref API-bmarray-bits</code> 00647 - <code>\ref API-bmarray-arange</code> 00648 - <code>\ref API-bmarray-brange</code> 00649 00650 \section API-bmarray-init init_bitmap_array(bmarray text, array_range text, bitmap_range text) 00651 \verbatim 00652 function veil.init_bitmap_array(bmarray text, array_range text, bitmap_range text) returns bool 00653 \endverbatim 00654 Creates or resets (clears) the bitmap array named <code>bmarray</code>. 00655 The last two parameters are the names of ranges used to bound the 00656 dimensions of the array, and the range of bits within the array's 00657 bitmaps. Implemented by C function veil_init_bitmap_array(). 00658 00659 \section API-bmarray-clear clear_bitmap_array(bmarray text) 00660 \verbatim 00661 function veil.clear_bitmap_array(bmarray text) returns bool 00662 \endverbatim 00663 Clear all bits in all bitmaps of the bitmap array named 00664 <code>bmarray</code>. Implemented by C function veil_clear_bitmap_array(). 00665 00666 \section API-bmarray-bmap bitmap_from_array(bmref_name text, bmarray text, index int4) 00667 \verbatim 00668 function veil.bitmap_from_array(bmref_name text, bmarray text, index int4) returns text 00669 \endverbatim 00670 Place a reference into <code>bmref_name</code> to the bitmap identified 00671 by <code>index</code> in bitmap array <code>bmarray</code>. Implemented 00672 by C function veil_bitmap_from_array(). 00673 00674 \section API-bmarray-testbit bitmap_array_testbit(bmarray text, arr_idx int4, bitno int4) 00675 \verbatim 00676 function veil.bitmap_array_testbit(bmarray text, arr_idx int4, bitno int4) returns bool 00677 \endverbatim 00678 Test a specific bit in a bitmap array. Implemented by C function 00679 veil_bitmap_array_testbit(). 00680 00681 \section API-bmarray-setbit bitmap_array_setbit(bmarray text, arr_idx int4, bitno int4) 00682 \verbatim 00683 function veil.bitmap_array_setbit(bmarray text, arr_idx int4, bitno int4) returns bool 00684 \endverbatim 00685 Set a specific bit in a bitmap array. Implemented by C function 00686 veil_bitmap_array_setbit(). 00687 00688 \section API-bmarray-clearbit bitmap_array_clearbit(bmarray text, arr_idx int4, bitno int4) 00689 \verbatim 00690 function veil.bitmap_array_clearbit(bmarray text, arr_idx int4, bitno int4) returns bool 00691 \endverbatim 00692 Clear a specific bit in a bitmap array. Implemented by C function 00693 veil_bitmap_array_clearbit(). 00694 00695 \section API-bmarray-union union_from_bitmap_array(bitmap text, bmarray text, arr_idx int4) 00696 \verbatim 00697 function veil.union_from_bitmap_array(bitmap text, bmarray text, arr_idx int4) returns bool 00698 \endverbatim 00699 Union a bitmap with a specified bitmap from an array, with the result in 00700 the bitmap. Implemented by C function 00701 veil_union_from_bitmap_array(). This is a faster shortcut for the 00702 following logical construction: 00703 00704 \verbatim 00705 veil.bitmap_union(<bitmap>, veil.bitmap_from_array(<bitmap_array>, <index>)) 00706 \endverbatim 00707 00708 \section API-bmarray-intersect intersect_from_bitmap_array(bitmap text, bmarray text, arr_idx int4) 00709 \verbatim 00710 function veil.intersect_from_bitmap_array(bitmap text, bmarray text, arr_idx int4) returns bool 00711 \endverbatim 00712 Intersect a bitmap with a specified bitmap from an array, with the result in 00713 the bitmap. Implemented by C function 00714 veil_intersect_from_bitmap_array(). This is a faster shortcut for the 00715 following logical construction: 00716 00717 \verbatim 00718 veil.bitmap_intersect(<bitmap>, veil.bitmap_from_array(<bitmap_array>,<index>)) 00719 \endverbatim 00720 00721 \section API-bmarray-bits bitmap_array_bits(bmarray text, arr_idx int4) 00722 \verbatim 00723 function veil.bitmap_array_bits(bmarray text, arr_idx int4) returns setof int4 00724 \endverbatim 00725 Show all bits in the specific bitmap within an array. This is primarily 00726 intended for interactive use when developing and debugging Veil-based 00727 systems. Implemented by C function veil_bitmap_array_bits(). 00728 00729 \section API-bmarray-arange bitmap_array_arange(bmarray text) 00730 \verbatim 00731 function veil.bitmap_array_arange(bmarray text) returns veil_range_t 00732 \endverbatim 00733 Return the range of array indices, as a \ref veil_range_t, for the 00734 specified bitmap array. Primarily for interactive use. Implemented by 00735 C function veil_bitmap_array_arange(). 00736 00737 \section API-bmarray-brange bitmap_array_brange(bmarray text) 00738 \verbatim 00739 function veil.bitmap_array_brange(bmarray text) returns veil_range_t 00740 \endverbatim 00741 Show the range, as a \ref veil_range_t, of all bitmaps in the specified 00742 bitmap array. Primarily for interactive use. Implemented by 00743 C function veil_bitmap_array_range(). 00744 00745 00746 Next: \ref API-bitmap-hashes 00747 */ 00748 /*! \page API-bitmap-hashes Bitmap Hashes 00749 A bitmap hashes is a hash table of identically-ranged bitmaps, indexed 00750 by a text key. 00751 00752 Typically bitmap hashes are used for sparse collections of privileges. 00753 00754 Note that bitmap hashes may not be stored in shared variables as hashes 00755 in shared memory are insufficiently dynamic. 00756 00757 The following functions comprise the Veil bitmap hashes API: 00758 00759 - <code>\ref API-bmhash-init</code> 00760 - <code>\ref API-bmhash-clear</code> 00761 - <code>\ref API-bmhash-key-exists</code> 00762 - <code>\ref API-bmhash-from</code> 00763 - <code>\ref API-bmhash-testbit</code> 00764 - <code>\ref API-bmhash-setbit</code> 00765 - <code>\ref API-bmhash-clearbit</code> 00766 - <code>\ref API-bmhash-union-into</code> 00767 - <code>\ref API-bmhash-union-from</code> 00768 - <code>\ref API-bmhash-intersect-from</code> 00769 - <code>\ref API-bmhash-bits</code> 00770 - <code>\ref API-bmhash-range</code> 00771 - <code>\ref API-bmhash-entries</code> 00772 00773 \section API-bmhash-init init_bitmap_hash(bmhash text, range text) 00774 \verbatim 00775 function veil.init_bitmap_hash(bmhash text, range text) returns bool 00776 \endverbatim 00777 Creates, or resets, a bitmap hash. Implemented by 00778 C function veil_init_bitmap_hash(). 00779 00780 \section API-bmhash-clear clear_bitmap_hash(bmhash text) 00781 \verbatim 00782 function veil.clear_bitmap_hash(bmhash text) returns bool 00783 \endverbatim 00784 Clear all bits in all bitmaps of a bitmap hash. Implemented by 00785 C function veil_clear_bitmap_hash(). Implemented by 00786 C function veil_clear_bitmap_hash(). 00787 00788 \section API-bmhash-key-exists bitmap_hash_key_exists(bmhash text, key text) 00789 \verbatim 00790 function veil.bitmap_hash_key_exists(bmhash text, key text) returns bool 00791 \endverbatim 00792 Determine whether a given key exists in the hash (contains a bitmap). 00793 Implemented by C function veil_bitmap_hash_key_exists(). 00794 00795 \section API-bmhash-from bitmap_from_hash(bmref text, bmhash text, key text) 00796 \verbatim 00797 function veil.bitmap_from_hash(bmref text, bmhash text, key text) returns text 00798 \endverbatim 00799 Generate a reference to a specific bitmap in a bitmap hash. Implemented by 00800 C function veil_bitmap_from_hash(). 00801 00802 \section API-bmhash-testbit bitmap_hash_testbit(bmhash text, key text, bitno int4) 00803 \verbatim 00804 function veil.bitmap_hash_testbit(bmhash text, key text, bitno int4) returns bool 00805 \endverbatim 00806 Test a specific bit in a bitmap hash. Implemented by 00807 C function veil_bitmap_hash_testbit(). 00808 00809 \section API-bmhash-setbit bitmap_hash_setbit(bmhash text, kay text, bitno int4) 00810 \verbatim 00811 function veil.bitmap_hash_setbit(bmhash text, key text, bitno int4) returns bool 00812 \endverbatim 00813 Set a specific bit in a bitmap hash. Implemented by 00814 C function veil_bitmap_hash_setbit(). 00815 00816 \section API-bmhash-clearbit bitmap_hash_clearbit(bmhash text, key text, bitno int4) 00817 \verbatim 00818 function veil.bitmap_hash_clearbit(bmhash text, key text, bitno int4) returns bool 00819 \endverbatim 00820 Clear a specific bit in a bitmap hash. Implemented by 00821 C function veil_bitmap_hash_clearbit(). 00822 00823 \section API-bmhash-union-into union_into_bitmap_hash(bmhash text, key text, bitmap text) 00824 \verbatim 00825 function veil.union_into_bitmap_hash(bmhash text, key text, bitmap text) returns bool 00826 \endverbatim 00827 Union a specified bitmap from a hash with a bitmap, with the result in 00828 the bitmap hash. Implemented by C function 00829 veil_union_into_bitmap_hash(). This is a faster shortcut for the 00830 following logical construction: 00831 00832 \verbatim 00833 veil.bitmap_union(veil.bitmap_from_hash(<bitmap_hash>, <key>), <bitmap>) 00834 \endverbatim 00835 00836 \section API-bmhash-union-from union_from_bitmap_hash(bmhash text, key text, bitmap text) 00837 \verbatim 00838 function veil.union_from_bitmap_hash(bmhash text, key text, bitmap text) returns bool 00839 \endverbatim 00840 Union a bitmap with a specified bitmap from a hash, with the result in 00841 the bitmap. Implemented by C function veil_union_from_bitmap_hash(). 00842 This is a faster shortcut for the following logical construction: 00843 00844 \verbatim 00845 veil.bitmap_union(<bitmap>, veil.bitmap_from_hash(<bitmap_array>, <key>)) 00846 \endverbatim 00847 00848 \section API-bmhash-intersect-from intersect_from_bitmap_hash(bitmap text, bmhash text, key text) 00849 \verbatim 00850 function veil.intersect_from_bitmap_hash(bitmap text, bmhash text, key text) returns bool 00851 \endverbatim 00852 Intersect a bitmap with a specified bitmap from a hash, with the result 00853 in the bitmap. Implemented by C function 00854 veil_intersect_from_bitmap_hash(). This is a faster shortcut for the 00855 following logical construction: 00856 00857 \verbatim 00858 veil.bitmap_intersect(<bitmap>, veil.bitmap_from_hash(<bitmap_array>, <key>)) 00859 \endverbatim 00860 00861 \section API-bmhash-bits bitmap_hash_bits(bmhash text, key text) 00862 \verbatim 00863 function veil.bitmap_hash_bits(bmhash text, key text) returns setof int4 00864 \endverbatim 00865 Show all bits in the specific bitmap within a hash. This is primarily 00866 intended for interactive use when developing and debugging Veil-based 00867 systems. Implemented by C function veil_bitmap_hash_bits(). 00868 00869 \section API-bmhash-range bitmap_hash_range(bmhash text) 00870 \verbatim 00871 function veil.bitmap_hash_range(bmhash text) returns veil_range_t 00872 \endverbatim 00873 Show the range, as a \ref veil_range_t, of all bitmaps in the hash. 00874 Primarily intended for interactive use. Implemented by 00875 C function veil_bitmap_hash_range(). 00876 00877 \section API-bmhash-entries bitmap_hash_entries(bmhash text) 00878 \verbatim 00879 function veil.bitmap_hash_entries(bmhash text) returns setof text 00880 \endverbatim 00881 Show every key in the hash. Primarily intended for interactive use. 00882 Implemented by C function veil_bitmap_hash_entries(). 00883 00884 Next: \ref API-int-arrays 00885 */ 00886 /*! \page API-int-arrays Integer Arrays 00887 Integer arrays are used to store simple mappings of keys to values. In 00888 the Veil demo (\ref demo-sec) they are used to record the extra privilege 00889 required to access person_details and project_details of each 00890 detail_type: the integer array being used to map the detail_type_id to 00891 the privilege_id. 00892 00893 Note that integer array elements cannot be null. 00894 00895 The following functions comprise the Veil int arrays API: 00896 00897 - <code>\ref API-intarray-init</code> 00898 - <code>\ref API-intarray-clear</code> 00899 - <code>\ref API-intarray-set</code> 00900 - <code>\ref API-intarray-get</code> 00901 00902 \section API-intarray-init init_int4array(arrayname text, range text) 00903 \verbatim 00904 function veil.init_int4array(arrayname text, range text) returns bool 00905 \endverbatim 00906 Creates, or resets the ranges of, an int array. Implemented by 00907 C function veil_init_int4array(). 00908 00909 \section API-intarray-clear clear_int4array(arrayname text) 00910 \verbatim 00911 function veil.clear_int4array(arrayname text) returns bool 00912 \endverbatim 00913 Clears (zeroes) an int array. Implemented by 00914 C function veil_clear_int4array(). 00915 00916 \section API-intarray-set int4array_set(arrayname text, idx int4, value int4) 00917 \verbatim 00918 function veil.int4array_set(arrayname text, idx int4, value int4) returns int4 00919 \endverbatim 00920 Set the value of an element in an int array. Implemented by 00921 C function veil_int4array_set(). 00922 00923 \section API-intarray-get int4array_get(arrayname text, idx int4) 00924 \verbatim 00925 function int4array_get(arrayname text, idx int4) returns int4 00926 \endverbatim 00927 Get the value of an element from an int array. Implemented by 00928 C function veil_int4array_get(). 00929 00930 Next: \ref API-serialisation 00931 */ 00932 /*! \page API-serialisation Veil Serialisation Functions 00933 With modern web-based applications, database connections are often 00934 pooled, with each connection representing many different users. In 00935 order to reduce the overhead of connection functions for such 00936 applications, Veil provides a serialisation API. This allows session 00937 variables for a connected user to be saved for subsequent re-use. This 00938 is particularly effective in combination with pgmemcache 00939 http://pgfoundry.org/projects/pgmemcache/ 00940 00941 Only session variables may be serialised. 00942 00943 The following functions comprise the Veil serialisatation API: 00944 00945 - <code>\ref API-serialise</code> 00946 - <code>\ref API-deserialise</code> 00947 - <code>\ref API-serialize</code> 00948 - <code>\ref API-deserialize</code> 00949 00950 \section API-serialise serialise(varname text) 00951 \verbatim 00952 function veil.serialise(varname text) returns text 00953 \endverbatim 00954 This creates a serialised textual representation of the named session 00955 variable. The results of this function may be concatenated into a 00956 single string, which can be deserialised in a single call to 00957 veil_deserialise(). Implemented by C function veil_serialise(). 00958 00959 \section API-deserialise deserialise(stream text) 00960 \verbatim 00961 function veil.deserialise(stream text) returns text 00962 \endverbatim 00963 This takes a serialised representation of one or more variables as 00964 created by concatenating the results of veil_serialise(), and 00965 de-serialises them, creating new variables as needed and resetting their 00966 values to those they had when they were serialised. Implemented by C 00967 function veil_deserialise(). 00968 00969 \section API-serialize serialize(varname text) 00970 \verbatim 00971 function veil.serialize(varname text) returns text 00972 \endverbatim 00973 Synonym for veil_serialise() 00974 00975 \section API-deserialize deserialize(stream text) 00976 \verbatim 00977 function veil.deserialize(stream text) returns text 00978 \endverbatim 00979 Synonym for veil_deserialise() 00980 00981 Next: \ref API-control 00982 */ 00983 /*! \page API-control Veil Control Functions 00984 Veil generally requires no management. The exception to this is when 00985 you wish to reset shared variables. You may wish to do this because 00986 your underlying security definitions have changed, or because you have 00987 added new features. In this case, you may use veil_perform_reset() to 00988 re-initialise your shared variables. This function replaces the current 00989 set of shared variables with a new set in a transaction-safe manner. 00990 All current transactions will complete with the old set of variables in 00991 place. All subsequent transactions will see the new set. 00992 00993 The following functions comprise the Veil control functions API: 00994 00995 - <code>\ref API-control-registered-init</code> 00996 - <code>\ref API-control-init</code> 00997 - <code>\ref API-control-reset</code> 00998 - <code>\ref API-version</code> 00999 01000 \section API-control-registered-init registered initialisation functions 01001 01002 A registered initialisation function is one which will be called from 01003 the standard veil \ref API-control-init function. Such functions are 01004 responsible for defining and, usually, initialising shared variables. 01005 01006 Initialisation functions may be written in any language supported by 01007 PostgreSQL, and must conform to the following function prototype: 01008 01009 \verbatim 01010 function init_function(doing_reset bool) returns bool 01011 \endverbatim 01012 01013 The <code>doing_reset</code> parameter will be set to true if we are 01014 completely resetting veil and redefining all of its variables. In this 01015 case, we must declare and, probably, initialise shared variables prior to 01016 any session initialisation actions. The parameter will be false, if the 01017 function is solely being called to initialise a new session. Check \ref 01018 demo-sec for an example. 01019 01020 Initialisation functions are registered by inserting their name into 01021 the configuration table <code>veil.veil_init_fns</code>. The functions 01022 listed in this table are executed in order of the <code>priority</code> 01023 column. Eg, to register <code>veil.init_roles()</code> to execute 01024 before <code>veil.init_role_privs()</code>, we would use the following 01025 insert statement: 01026 01027 \verbatim 01028 insert into veil.veil_init_fns 01029 (fn_name, priority) 01030 values ('veil.init_roles', 1), 01031 ('veil.init_role_privs', 2); 01032 \endverbatim 01033 01034 \section API-control-init veil_init(doing_reset bool) 01035 \verbatim 01036 function veil.veil_init(doing_reset bool) returns bool 01037 \endverbatim 01038 01039 This function, implemented by the C function veil_init(), is reponsible 01040 for initialising each veil session. The <code>doing_reset</code> 01041 parameter is true if we are to completely reset Veil, redefining all 01042 shared variables. 01043 01044 The builtin implementation of veil_init() will call each registered 01045 initialisation function (see \ref API-control-registered-init) in turn. 01046 01047 If no initialisation functions are registered, veil_init() raises an 01048 exception. 01049 01050 As an alternative to registering initialisation functions, a Veil-based 01051 application may instead simply redefine veil.veil_init(), though this 01052 usage is deprecated. 01053 01054 \section API-control-reset veil_perform_reset() 01055 \verbatim 01056 function veil.veil_perform_reset() returns bool 01057 \endverbatim 01058 This is used to reset Veil's shared variables. It causes \ref 01059 API-control-init to be called. Implemented by C function veil_perform_reset(). 01060 01061 \section API-version version() 01062 \verbatim 01063 function veil.version() returns text 01064 \endverbatim 01065 This function returns a string describing the installed version of 01066 veil. Implemented by C function veil_version(). 01067 01068 Next: \ref Building 01069 01070 */ 01071 /*! \page Building Building a Veil-based secure database 01072 \section Build-sec Building a Veil-based secure database 01073 01074 This section describes the steps necessary to secure a database using 01075 Veil. The steps are: 01076 - \ref Policy 01077 - \ref Schemas 01078 - \ref Design 01079 - \ref Implementation 01080 - \ref Implementation2 01081 - \ref Implementation3 01082 - \ref Implementation4 01083 - \ref Testing 01084 01085 \subsection Policy Determine your Policies 01086 01087 You must identify which security contexts exist for your application, 01088 and how privileges should be assigned to users in those contexts. You 01089 must also figure out how privileges, roles, and the assignment of roles 01090 to users are to be managed. You must identify each object that is to be 01091 protected by Veil, identify the security contexts applicable for that 01092 object, and determine the privileges that will apply to each object in 01093 each possible mode of use. Use the Veil demo application (\ref 01094 demo-sec) as a guide. 01095 01096 For data access controls, typically you will want specific privileges 01097 for select, insert, update and delete on each table. You may also want 01098 separate admin privileges that allow you to grant those rights. 01099 01100 At the functional level, you will probably have an execute privilege for 01101 each callable function, and you will probably want similar privileges 01102 for individual applications and components of applications. Eg, to 01103 allow the user to execute the role_manager component of admintool, you 01104 would probably create a privilege called 01105 <code>exec_admintool_roleman</code>. 01106 01107 The hardest part of this is figuring out how you will securely manage 01108 these privileges. A useful, minimal policy is to not allow anyone to 01109 assign a role that they themselves have not been assigned. 01110 01111 \subsection Schemas Design Your Database-Level Security 01112 01113 Veil operates within the security provided by PostgreSQL. If you wish 01114 to use Veil to protect underlying tables, then those tables must not be 01115 directly accessible to the user. Also, the Veil functions themselves, 01116 as they provide privileged operations, must not be accessible to user 01117 accounts. 01118 01119 A sensible basic division of schema responsibilities would be as follows: 01120 01121 - An "owner" user will own the underlying objects (tables, views, 01122 functions, etc) that are to be secured. Access to these objects will 01123 be granted only to "Veil". The "owner" user will connect only when 01124 the underlying objects are to be modified. No-one but a DBA will ever 01125 connect to this account, and generally, the password for this account 01126 should be disabled. 01127 01128 - A "Veil" user will own all secured views and access functions (see 01129 \ref over-views). Access to these objects will be granted to the 01130 "Accessor" user. Like the "owner" user, this user should not be 01131 directly used except by DBAs performing maintenance. It will also own 01132 the Veil API, ie this is the account where Veil itself will be 01133 installed. Direct access to Veil API functions should not be granted 01134 to other users. If access to a specific function is needed, it should 01135 be wrapped in a local function to which access may then be granted. 01136 01137 - "Accessor" users are the primary point of contact. These must have no 01138 direct access to the underlying objects owned by owner. They will have 01139 access only to the secured views and access functions. All 01140 applications may connect to these user accounts. 01141 01142 \subsection Design Design Your Access Functions 01143 01144 Provide a high-level view of the workings of each access function. You 01145 will need this in order to figure out what session and shared variables 01146 you will need. The following is part of the design from the Veil demo 01147 application: 01148 \verbatim 01149 Access Functions are required for: 01150 - Global context only (lookup-data, eg privileges, roles, etc) 01151 - Personal and Global Context (personal data, persons, assignments, etc) 01152 - Project and Global (projects, project_details) 01153 - All 3 (assignments) 01154 01155 Determining privilege in Global Context: 01156 01157 User has priv X, if X is in role_privileges for any role R, that has 01158 been assigned to the user. 01159 01160 Role privileges are essentially static so may be loaded into memory as a 01161 shared variable. When the user connects, the privileges associated with 01162 their roles may be loaded into a session variable. 01163 01164 Shared initialisation code: 01165 role_privs ::= shared array of privilege bitmaps indexed by role. 01166 Populate role_privs with: 01167 select bitmap_array_setbit(role_privs, role_id, privilege_id) 01168 from role_privileges; 01169 01170 Connection initialisation code: 01171 global_privs ::= session privileges bitmap 01172 Clear global_privs and then initialise with: 01173 select bitmap_union(global_privs, role_privs[role_id]) 01174 from person_roles 01175 where person_id = connected_user; 01176 01177 i_have_global_priv(x): 01178 return bitmap_testbit(global_privs, x); 01179 01180 \endverbatim 01181 01182 This gives us the basic structure of each function, and identifies what 01183 must be provided by session and system initialisation to support those 01184 functions. It also allows us to identify the overhead that Veil imposes. 01185 01186 In the case above, there is a connect-time overhead of one extra query 01187 to load the global_privs bitmap. This is probably a quite acceptable 01188 overhead as typically each user will have relatively few roles. 01189 01190 If the overhead of any of this seems too significant there are 01191 essentially 4 options: 01192 - Simplify the design. 01193 - Defer the overhead until it is absolutely necessary. This can be done 01194 with connection functions where we may be able to defer the overhead 01195 of loading relational context data until the time that we first need 01196 it. 01197 - Implement a caching solution (check out pgmemcache). Using an 01198 in-memory cache will save data set-up queries from having to be 01199 repeated. This is pretty complex though and may require you to write 01200 code in C. 01201 - Suffer the performance hit. 01202 01203 \subsection Implementation Implement the Initialisation Function 01204 01205 Proper initialisation of veil is critical. There are two ways to manage 01206 this. The traditional way is to write your own version of \ref 01207 API-control-init, replacing the supplied version. The newer, better, 01208 alternative is to register your own initialisation functions in the 01209 table veil.veil_init_fns, and have the standard \ref API-control-init, 01210 call them. If there are multiple initialisation functions, they are 01211 called in order of their priority values as specified in the table 01212 <code>veil.veil_init_fns</code>. 01213 01214 The newer approach has a number of advantages: 01215 01216 - it fully supports the PostgreSQL extension mechanism, allowing 01217 extensions to be created and dropped; 01218 - it allows different security subsystems to have their own separate 01219 initialisation routines, allowing more modular code and better 01220 separation of responsibilities; 01221 - it is way cooler. 01222 01223 Initialisation functions \ref API-control-init are critical elements. 01224 They will be called by automatically by Veil, when the first in-built 01225 Veil function is invoked. Initialisation functions are responsible for 01226 three distinct tasks: 01227 01228 - Initialisation of session variables 01229 - Initialisation of shared variables 01230 - Re-initialisation of variables during reset 01231 01232 The boolean parameter to veil_init (which is passed to registered 01233 initialisation functions) will be false on initial session 01234 startup, and true when performing a reset (\ref API-control-reset). 01235 01236 Shared variables are created using \ref API-variables-share. This 01237 returns a boolean result describing whether the variable already 01238 existed. If so, and we are not performing a reset, the current session 01239 need not initialise it. 01240 01241 Session variables are simply created by using them. It is worth 01242 creating and initialising all session variables to "fix" their data 01243 types. This will prevent other functions from misusing them. 01244 01245 If the boolean parameter to an initialisation fuction is true, then we 01246 are performing a memory reset, and all shared variables should be 01247 re-initialised. A memory reset will be performed whenever underlying, 01248 essentially static, data has been modified. For example, when new 01249 privileges have been added, we must rebuild all privilege bitmaps to 01250 accommodate the new values. 01251 01252 \subsection Implementation2 Implement the Connection Functions 01253 01254 The connection functions have to authenticate the connecting user, and 01255 then initialise the user's session. 01256 01257 Authentication should use a secure process in which no plaintext 01258 passwords are ever sent across the wire. Veil does not provide 01259 authentication services. For your security needs you should probably 01260 check out pgcrypto. 01261 01262 Initialising a user session is generally a matter of initialising 01263 bitmaps that describe the user's base privileges, and may also involve 01264 setting up bitmap hashes of their relational privileges. Take a look at 01265 the demo (\ref demo-sec) for a working example of this. 01266 01267 \subsection Implementation3 Implement the Access Functions 01268 01269 Access functions provide the low-level access controls to individual 01270 records. As such, their performance is critical. It is generally better 01271 to make the connection functions to more work, and the access functions 01272 less. Bear in mind that if you perform a query that returns 10,000 rows 01273 from a table, your access function for that view is going to be called 01274 10,000 times. It must be as fast as possible. 01275 01276 When dealing with relational contexts, it is not always possible to keep 01277 all privileges for every conceivable relationship in memory. When this 01278 happens, your access function will have to perform a query itself to 01279 load the specific data into memory. If your application requires this, 01280 you should: 01281 01282 - Ensure that each such query is as simple and efficient as possible 01283 - Cache your results in some way 01284 01285 You may be able to trade-off between the overhead of connection 01286 functions and that of access functions. For instance if you have a 01287 relational security context based upon a tree of relationships, you may 01288 be able to load all but the lowest level branches of the tree at connect 01289 time. The access function then has only to load the lowest level branch 01290 of data at access time, rather than having to perform a full tree-walk. 01291 01292 Caching can be very effective, particularly for nested loop joins. If 01293 you are joining A with B, and they both have the same access rules, once 01294 the necessary privilege to access a record in A has been determined and 01295 cached, we will be able to use the cached privileges when checking for 01296 matching records in B (ie we can avoid repeating the fetch). 01297 01298 \subsection Implementation4 Implement the views and instead-of triggers 01299 01300 This is the final stage of implementation. For every base table you 01301 must create a secured view and a set of instead-of triggers for insert, 01302 update and delete. Refer to the demo (\ref demo-sec) for details of 01303 this. 01304 01305 \subsection Testing Testing 01306 01307 Be sure to test it all. Specifically, test to ensure that failed 01308 connections do not provide any privileges, and to ensure that all 01309 privileges assigned to highly privileged users are cleared when a more 01310 lowly privileged user takes over a connection. Also ensure that 01311 the underlying tables and raw veil functions are not accessible from 01312 user accounts. 01313 01314 \section Automation Automatic code generation 01315 01316 Note that the bulk of the code in a Veil application is in the 01317 definition of secured views and instead-of triggers, and that this code 01318 is all very similar. Consider using a code-generation tool to implement 01319 this. 01320 01321 Next: \ref Demo 01322 01323 */ 01324 /*! \page Demo A Full Example Application: The Veil Demo 01325 \section demo-sec The Veil Demo Application 01326 01327 The Veil demo application serves two purposes: 01328 - it provides a demonstration of Veil-based access controls; 01329 - it provides a working example of how to build a secured system using Veil. 01330 01331 This section covers the following topics: 01332 01333 - \ref demo-install 01334 - \subpage demo-model 01335 - \subpage demo-security 01336 - \subpage demo-explore 01337 - \subpage demo-code 01338 - \subpage demo-uninstall 01339 01340 \subsection demo-install Installing the Veil demo 01341 01342 The veil_demo application, is packaged as an extension just like Veil 01343 itself, and is installed as part of the installation of Veil. To create 01344 a database containing the veil_demo application: 01345 - create a new database 01346 - connect to that database 01347 - execute <code>create extension veil;</code> 01348 - execute <code>create extension veil_demo;</code> 01349 01350 Next: \ref demo-model 01351 01352 */ 01353 /*! \page demo-model The Demo Database ERD, Tables and Views 01354 \section demo-erd The Demo Database ERD 01355 01356 \image html veil_demo.png "The Veil Demo Database" width=10cm 01357 01358 \section demo-tables Table Descriptions 01359 01360 \subsection demo-privs Privileges 01361 01362 This table describes each privilege. A privilege is a right to do 01363 something. Most privileges are concerned with providing access to 01364 data. For each table, "X" there are 4 data privileges, SELECT_X, UPDATE_X, 01365 INSERT_X and DELETE_X. There are separate privileges to provide access 01366 to project and person details, and there is a single function privilege, 01367 <code>can_connect</code>. 01368 01369 \subsection demo-roles Roles 01370 01371 A role is a named collection of privileges. Privileges are assigned to 01372 roles through role_privileges. Roles exist to reduce the number of 01373 individual privileges that have to be assigned to users, etc. Instead 01374 of assigning twenty or more privileges, we assign a single role that 01375 contains those privileges. 01376 01377 In this application there is a special role, <code>Personal 01378 Context</code> that contains the set of privileges that apply to all 01379 users in their personal context. Since all users have the same personal 01380 context privileges, the demo application provides this role to all users 01381 implicitly; there is no need for it to be explicitly assigned. 01382 01383 Assignments of roles in the global context are made through 01384 person_roles, and in the project (relational) context through 01385 assignments. 01386 01387 \subsection demo-role-privs Role_Privileges 01388 01389 Role privileges describe the set of privileges for each role. 01390 01391 \subsection demo-role-roles Role_Roles 01392 01393 This is currently unused in the Veil demo application. Role roles 01394 provides the means to assign roles to other roles. This allows new 01395 roles to be created as combinations of existing roles. The use of this 01396 table is currently left as an exercise for the reader. 01397 01398 \subsection demo-persons Persons 01399 01400 This describes each person. A person is someone who owns data and who 01401 may connect to the database. This table should contain authentication 01402 information etc. In the demo it just maps a name to a person_id. 01403 01404 \subsection demo-projects Projects 01405 01406 A project represents a real-world project, to which many persons may be 01407 assigned to work. 01408 01409 \subsection demo-person-roles Person_Roles 01410 01411 This table describes the which roles have been assigned to users in the 01412 global context. 01413 01414 \subsection demo-assignments Assignments 01415 01416 This describes the roles that have been assigned to a person on a 01417 specific project. Assignments provide privilege to a user in the 01418 project context. 01419 01420 \subsection demo-detail_types Detail_Types 01421 01422 This is a lookup-table that describes general-purpose attributes that 01423 may be assigned to persons or project. An example of an attribute for a 01424 person might be birth-date. For a project it might be inception date. 01425 This allows new attributes to be recorded for persons, projects, etc 01426 without having to add columns to the table. 01427 01428 Each detail_type has a required_privilege field. This identifies the 01429 privilege that a user must have in order to be able to see attributes of 01430 the specific type. 01431 01432 \subsection demo-person_details Person_Details 01433 01434 These are instances of specific attributes for specific persons. 01435 01436 \subsection demo-project-details Project_Details 01437 01438 These are instances of specific attributes for specific projects. 01439 01440 \section Demo-Views The Demo Application's Helper Views 01441 01442 Getting security right is difficult. The Veil demo provides a number of 01443 views that help you view the privileges you have in each context. 01444 01445 - my_global_privs shows you the privileges you have in the global 01446 context 01447 - my_personal_privs shows you the privileges you have in the 01448 personal context 01449 - my_project_privs shows you the privileges you have for each project 01450 in the project context 01451 - my_privs shows you all your privileges in all contexts 01452 - my_projects shows you all the projects to which you have been assigned 01453 01454 Using these views, access control mysteries may be more easily tracked 01455 down. 01456 01457 Next: \ref demo-security 01458 01459 */ 01460 /*! \page demo-security The Demo Database Security Model 01461 \section demo-secmodel The Demo Database Security Model 01462 01463 The Veil demo has three security contexts. 01464 01465 - Personal Context applies to personal data that is owned by the 01466 connected user. All users have the same privileges in personal 01467 context, as defined by the role <code>Personal Context</code>. 01468 - Global Context applies equally to every record in a table. If a user 01469 has <code>SELECT_X</code> privilege in the global context, they will 01470 be able to select every record in <code>X</code>, regardless of 01471 ownership. Privileges in global context are assigned through 01472 <code>person_roles</code>. 01473 - Project Context is a relational context and applies to project data. 01474 If you are assigned a role on a project, you will be given specific 01475 access to certain project tables. The roles you have been assigned 01476 will define your access rights. 01477 01478 The following sections identify which tables may be accessed in which 01479 contexts. 01480 01481 \subsection demo-global-context The Global Context 01482 The global context applies to all tables. All privilege checking 01483 functions will always look for privileges in the global context. 01484 01485 \subsection demo-personal-context Personal Context 01486 The following tables may be accessed using rights assigned in the 01487 personal context: 01488 - persons 01489 - assignments 01490 - person_details 01491 01492 \subsubsection demo-project-context Project Context 01493 The following tables may be accessed using rights assigned in the 01494 project context: 01495 - projects 01496 - assignments 01497 - project_details 01498 01499 Next: \ref demo-explore 01500 01501 */ 01502 /*! \page demo-explore Exploring the Demo 01503 \section demo-use Exploring the Demo 01504 \subsection demo-connect Accessing the Demo Database 01505 Using your favourite tool connect to your veil_demo database. 01506 01507 You will be able to see all of the demo views, both the secured views and 01508 the helpers. But you will not initially be able to see any records: 01509 each view will appear to contain no data. To gain some privileges you 01510 must identify yourself using the \ref demo-code-connect-person function. 01511 01512 There are 6 persons in the demo. You may connect as any of them and see 01513 different subsets of data. The persons are 01514 01515 - 1 Deb (the DBA). Deb has global privileges on everything. She needs 01516 them as she is the DBA. 01517 - 2 Pat (the PM). Pat has the manager role globally, and is the project 01518 manager of project 102. Pat can see all but the most confidential 01519 personal data, and all data about her project. 01520 - 3 Derick (the director). Derick can see all personal and project 01521 data. He is also the project manager for project 101, the secret 01522 project. 01523 - 4 Will (the worker). Will has been assigned to both projects. He has 01524 minimal privileges and cannot access project confidential data. 01525 - 5 Wilma (the worker). Willma has been assigned to project 101. She has 01526 minimal privileges and cannot access project confidential data. 01527 - 6 Fred (the fired DBA). Fred has all of the privileges of Deb, except 01528 for can_connect privilege. This prevents Fred from being able to do 01529 anything. 01530 01531 Here is a sample session, showing the different access enjoyed by 01532 different users. 01533 01534 \verbatim 01535 veildemo=> select connect_person(4); 01536 connect_person 01537 ---------------- 01538 t 01539 (1 row) 01540 01541 veildemo=> select * from persons; 01542 person_id | person_name 01543 -----------+------------------- 01544 4 | Will (the worker) 01545 (1 row) 01546 01547 veildemo=> select * from person_details; 01548 person_id | detail_type_id | value 01549 -----------+----------------+-------------- 01550 4 | 1003 | 20050105 01551 4 | 1002 | Employee 01552 4 | 1004 | 30,000 01553 4 | 1005 | 19660102 01554 4 | 1006 | 123456789 01555 4 | 1007 | Subservience 01556 (6 rows) 01557 01558 veildemo=> select * from project_details; 01559 project_id | detail_type_id | value 01560 ------------+----------------+---------- 01561 102 | 1001 | 20050101 01562 102 | 1002 | Ongoing 01563 (2 rows) 01564 01565 veildemo=> select connect_person(2); 01566 connect_person 01567 ---------------- 01568 t 01569 (1 row) 01570 01571 veildemo=> select * from person_details; 01572 person_id | detail_type_id | value 01573 -----------+----------------+------------------- 01574 1 | 1003 | 20050102 01575 2 | 1003 | 20050103 01576 3 | 1003 | 20050104 01577 4 | 1003 | 20050105 01578 5 | 1003 | 20050106 01579 6 | 1003 | 20050107 01580 1 | 1002 | Employee 01581 2 | 1002 | Employee 01582 3 | 1002 | Employee 01583 4 | 1002 | Employee 01584 5 | 1002 | Employee 01585 6 | 1002 | Terminated 01586 2 | 1004 | 50,000 01587 1 | 1005 | 19610102 01588 2 | 1005 | 19600102 01589 3 | 1005 | 19650102 01590 4 | 1005 | 19660102 01591 5 | 1005 | 19670102 01592 2 | 1006 | 123456789 01593 1 | 1007 | Oracle, C, SQL 01594 2 | 1007 | Soft peoply-stuff 01595 3 | 1007 | None at all 01596 4 | 1007 | Subservience 01597 5 | 1007 | Subservience 01598 (24 rows) 01599 01600 veildemo=> select * from project_details; 01601 project_id | detail_type_id | value 01602 ------------+----------------+---------- 01603 102 | 1001 | 20050101 01604 102 | 1002 | Ongoing 01605 102 | 1008 | $100,000 01606 (3 rows) 01607 01608 veildemo=> 01609 01610 \endverbatim 01611 01612 Next: \ref demo-code 01613 01614 */ 01615 /*! \page demo-code The Demo Code 01616 \section demo-codesec The Code 01617 \subsection demo-code-veil-init veil.veil_demo_init(performing_reset bool) 01618 01619 This function is called at the start of each session, and whenever 01620 \ref API-control-reset is called. The parameter, doing_reset, is 01621 false when called to initialise a session and true when called from 01622 veil_perform_reset(). It is registered with \ref API-control-init 01623 through the <code>veil.veil_demo_init_fns</code> table which is created 01624 as an inherited table of <code>veil.veil_init_fns</code>. By 01625 registering the initialisation functions using a veil_demo-specific 01626 inherited table, when the veil_demo extension is dropped, so is the 01627 registration data for \ref demo-code-veil-init. 01628 01629 \dontinclude veil_demo.sqs 01630 \skip veil_demo_init(doing 01631 \until init_reqd = 01632 01633 The first task of veil_init() is to declare a set of Veil shared 01634 variables. This is done by calling \ref API-variables-share. This function 01635 returns true if the variable already exists, and creates the variable 01636 and returns false, if not. 01637 01638 These variables are defined as shared because they will be identical for 01639 each session. Making them shared means that only one session has to 01640 deal with the overhead of their initialisation. 01641 01642 \dontinclude veil_demo.sqs 01643 \skip init_reqd = 01644 \until end if; 01645 01646 We then check whether the shared variables must be initialised. We will 01647 initialise them if they have not already been initialised by another 01648 session, or if we are performing a reset (see \ref API-control-reset). 01649 01650 Each variable is initialised in its own way. 01651 01652 Ranges are set by a single call to \ref API-basic-init-range. Ranges are 01653 used to create bitmap and array types of a suitable size. 01654 01655 Int4Arrays are used to record mappings of one integer to another. In 01656 the demo, they are used to record the mapping of detail_type_id to 01657 required_privilege_id. We use this variable so that we can look-up the 01658 privilege required to access a given project_detail or person_detail 01659 without having to explicitly fetch from attribute_detail_types. 01660 01661 Int4Arrays are initialised by a call to \ref API-intarray-init, and 01662 are populated by calling \ref API-intarray-set for each value to 01663 be recorded. Note that rather than using a cursor to loop through each 01664 detail_type record, we use select count(). This requires less code and 01665 has the same effect. 01666 01667 We use a BitmapArray to record the set of privileges for each role. Its 01668 initialisation and population is handled in much the same way as 01669 described above for Int4Arrays, using the functions \ref 01670 API-bmarray-init and \ref API-bmarray-setbit. 01671 01672 \until language 01673 01674 The final section of code defines and initialises a set of session 01675 variables. These are defined here to avoid getting undefined variable 01676 errors from any access function that may be called before an 01677 authenticated connection has been established. 01678 01679 Note that this and all Veil related functions are defined with 01680 <code>security definer</code> attributes. This means that the function 01681 will be executed with the privileges of the function's owner, rather 01682 than those of the invoker. This is absolutely critical as the invoker 01683 must have no privileges on the base objects, or on the raw Veil 01684 functions themselves. The only access to objects protected by Veil must 01685 be through user-defined functions and views. 01686 01687 \subsection demo-code-connect-person connect_person(_person_id int4) 01688 01689 This function is used to establish a connection from a specific person. 01690 In a real application this function would be provided with some form of 01691 authentication token for the user. For the sake of simplicity the demo 01692 allows unauthenticated connection requests. 01693 01694 \skip connect_person(_per 01695 \until language 01696 01697 This function identifies the user, ensures that they have can_connect 01698 privilege. It initialises the global_context bitmap to contain the 01699 union of all privileges for each role the person is assigned through 01700 person_roles. It also sets up a bitmap hash containing a bitmap of 01701 privileges for each project to which the person is assigned. 01702 01703 \subsection demo-code-global-priv i_have_global_priv(priv_id int4) 01704 01705 This function is used to determine whether a user has a specified 01706 privilege in the global context. It tests that the user is connected 01707 using \ref API-basic-int4-get, and then checks whether the 01708 specified privilege is present in the <code>global_context</code> 01709 bitmap. 01710 01711 \skip function i_have_global_priv(priv 01712 \until security definer; 01713 01714 The following example shows this function in use by the secured view, 01715 <code>privileges</code>: 01716 01717 \skip create view privileges 01718 \until grant 01719 01720 The privileges used above are <code>select_privileges</code> (10001), 01721 <code>insert_privileges</code> (10002), <code>update_privileges</code> 01722 (10003), and <code>delete_privileges</code> (10004). 01723 01724 \subsection demo-code-personal-priv i_have_personal_priv(priv_id int4, person_id int4) 01725 01726 This function determines whether a user has a specified privilege to a 01727 specified user's data, in the global or personal contexts. It performs 01728 the same tests as for \ref demo-code-global-priv. If the user 01729 does not have access in the global context, and the connected user is 01730 the same user as the owner of the data we are looking at, then we test 01731 whether the specified privilege exists in the <code>role_privs</code> 01732 bitmap array for the <code>Personal Context</code> role. 01733 01734 \dontinclude veil_demo.sqs 01735 \skip function i_have_personal_priv(pr 01736 \until language 01737 01738 Here is an example of this function in use from the persons secured view: 01739 01740 \skip create view persons 01741 \until grant 01742 01743 \subsection demo-code-project-priv i_have_project_priv(priv_id int4, project_id int4) 01744 This function determines whether a user has a specified privilege in the 01745 global or project contexts. If the user does not have the global 01746 privilege, we check whether they have the privilege defined in the 01747 project_context BitmapHash. 01748 01749 \dontinclude veil_demo.sqs 01750 \skip function i_have_project_priv(pr 01751 \until language 01752 01753 Here is an example of this function in use from the instead-of insert 01754 trigger for the projects secured view: 01755 01756 \skip create rule ii_projects 01757 \until i_have_project_priv(10018, new.project_id); 01758 01759 \subsection demo-code-proj-pers-priv i_have_proj_or_pers_priv(priv_id int4, project_id int4, person_id int4) 01760 This function checks all privileges. It starts with the cheapest check 01761 first, and short-circuits as soon as a privilege is found. 01762 01763 \dontinclude veil_demo.sqs 01764 \skip function i_have_proj_or_pers_priv( 01765 \until language 01766 01767 Here is an example of this function in use from the instead-of update 01768 trigger for the assignments secured view: 01769 01770 \skip create rule ii_assignments 01771 \until i_have_proj_or_pers_priv(10027, old.project_id, old.person_id); 01772 01773 \subsection demo-code-pers-detail-priv i_have_person_detail_priv(detail_id int4, person_id int4) 01774 This function is used to determine which types of person details are 01775 accessible to each user. This provides distinct access controls to each 01776 attribute that may be recorded for a person. 01777 01778 \dontinclude veil_demo.sqs 01779 \skip function i_have_person_detail_priv( 01780 \until language 01781 01782 The function is shown in use, below, in the instead-of delete trigger 01783 for person_details. Note that two distinct access functions are being 01784 used here. 01785 01786 \skip create rule id_person_details 01787 \until i_have_person_detail_priv(old.detail_type_id, old.person_id); 01788 01789 Next: \ref demo-uninstall 01790 01791 */ 01792 /*! \page demo-uninstall Removing The Demo Database 01793 \section demo-clean-up Removing The Demo Database 01794 In your veil_demo database execute: 01795 - <code>drop extension veil_demo;</code> 01796 01797 Next: \ref Management 01798 01799 */ 01800 /*! \page Management Managing Privileges, etc 01801 \section Management-sec Managing Privileges, etc 01802 The management of privileges and their assignments to roles, persons, 01803 etc are the key to securing a veil-based application. It is therefore 01804 vital that privilege assignment is itself a privileged operation. 01805 01806 The veil demo does not provide an example of how to do this, and this 01807 section does little more than raise the issue. 01808 01809 IT IS VITAL THAT YOU CAREFULLY LIMIT HOW PRIVILEGES ARE MANIPULATED AND 01810 ASSIGNED! 01811 01812 Here are some possible rules of thumb that you may wish to apply: 01813 01814 - give only the most senior and trusted users the ability to assign 01815 privileges; 01816 - allow only the DBAs to create privileges; 01817 - allow only 1 or 2 security administrators to manage roles; 01818 - allow roles or privileges to be assigned only by users that have both 01819 the "assign_privileges"/"assign_roles" privileges, and that themselves 01820 have the privilege or role they are assigning; 01821 - consider having an admin privilege for each table and only allow users 01822 to assign privileges on X if they have "admin_x" privilege; 01823 - limit the users who have access to the role/privilege management 01824 functions, and use function-level privileges to enforce this; 01825 - audit/log all assignments of privileges and roles; 01826 - send email to the security administrator whenever role_privileges are 01827 manipulated and when roles granting high-level privileges are granted. 01828 01829 Next: \ref Esoteria 01830 01831 */ 01832 /*! \page Esoteria Exotic and Esoteric uses of Veil 01833 01834 \section Esoteria-sec Exotic and Esoteric uses of Veil 01835 Actually this is neither exotic nor particularly esoteric. The title is 01836 simply wishful thinking on the author's part. 01837 \subsection layered-sessions Multi-Layered Connections 01838 So far we have considered access controls based only on the user. If we 01839 wish to be more paranoid, and perhaps we should, we may also consider 01840 limiting the access rights of each application. 01841 01842 This might mean that reporting applications would have no ability to 01843 update data, that financial applications would have no access to 01844 personnel data, and that personnel apps would have no access to business 01845 data. 01846 01847 This can be done in addition to the user-level checks, so that even if I 01848 have DBA privilege, I can not subvert the personnel reporting tools to 01849 modify financial data. 01850 01851 All access functions would check the service's privileges in addition to 01852 the user's before allowing any operation. 01853 01854 This could be implemented with a connect_service() function that would 01855 be called only once per session and that *must* be called prior to 01856 connecting any users. Alternatively, the connected service could be 01857 inferred from the account to which the service is connected. 01858 01859 \subsection columns Column-Level Access Controls 01860 01861 Although veil is primarily intended for row-based access controls, 01862 column-based is also possible. If this is required it may be better to 01863 use a highly normalised data model where columns are converted instead 01864 into attributes, much like the person_details and project_details tables 01865 from the demo application (\ref demo-sec). 01866 01867 If this is not possible then defining access_views that only show 01868 certain columns can be done something like this: 01869 01870 \verbatim 01871 create view wibble(key, col1, col2, col3) as 01872 select key, 01873 case when have_col_priv(100001) then col1 else null end, 01874 case when have_col_priv(100002) then col2 else null end, 01875 case when have_col_priv(100003) then col3 else null end 01876 where have_row_priv(1000); 01877 \endverbatim 01878 01879 The instead-of triggers for this are left as an exercise. 01880 01881 Next: \ref install 01882 01883 */ 01884 /*! \page install Installation and Configuration 01885 \section install_sec Installation 01886 \subsection Get Getting Veil 01887 Veil can be downloaded as a gzipped tarball from 01888 http://pgfoundry.org/projects/veil/ 01889 01890 Since version 9.1, git is no longer available from cvs on pgfoundry. 01891 Development has switched to using git. The primary git repository is 01892 git://github.com/marcmunro/veil.git 01893 01894 To checkout from git, create a suitable directory and do: 01895 01896 \verbatim 01897 git clone git://github.com/marcmunro/veil.git 01898 \endverbatim 01899 01900 An alternative repository is also available here: 01901 git@bloodnok.com:veil.git 01902 01903 \subsection Pre-requisites Pre-requisites 01904 You must have a copy of the Postgresql header files available in order 01905 to build Veil. For this, you may need to install the postgres developer 01906 packages for your OS, or even build postgres from source. 01907 \subsection build-sub Building Veil 01908 Veil is built using the postgresql extension building infastructure, 01909 PGXS. In order to be able to build using PGXS, the following command 01910 must return the location of the PGXS makefile: 01911 \verbatim 01912 $ pg_config --pgxs 01913 \endverbatim 01914 You may need to modify your PATH variable to make pg_config usable, or 01915 ensure it is for the correct postgresql version. 01916 01917 To build the postgres extensions, simply run make with no target: 01918 \verbatim 01919 $ make 01920 \endverbatim 01921 01922 To build the veil documentation (the documentation you are now reading) 01923 use make docs. 01924 01925 Note that the build system deliberately avoids using make recursively. 01926 Search the Web for "Recursive Make Considered Harmful" for the reasons 01927 why. This makes the construction of the build system a little different 01928 from what you may be used to. This may or may not turn out to be a good 01929 thing. \ref Feedback "Feedback" is welcomed. 01930 01931 \subsection Install Installing Veil 01932 As the postgres, pgsql, or root user, run <code>make install</code>. 01933 \verbatim 01934 make install 01935 \endverbatim 01936 01937 \section configuration Configuration 01938 To configure Veil, the following lines should be added to your 01939 postgresql.conf: 01940 \code 01941 shared_preload_libraries = '<path to shared library>/veil.so' 01942 01943 custom_variable_classes = 'veil' 01944 01945 #veil.dbs_in_cluster = 1 01946 #veil.shared_hash_elems = 32 01947 #veil.shmem_context_size = 16384 01948 \endcode 01949 01950 The three configuration options, commented out above, are: 01951 - dbs_in_cluster 01952 The number of databases, within the database cluster, that 01953 will use Veil. Each such database will be allocated 2 chunks of 01954 shared memory (of shmem_context_size), and a single LWLock. 01955 It defaults to 1. 01956 01957 - shared_hash_elems 01958 This describes how large a hash table should be created for veil 01959 shared variables. It defaults to 32. If you have more than about 20 01960 shared variables you may want to increase this to improve 01961 performance. This setting does not limit the number of variables that 01962 may be defined, it just limits how efficiently they may be accessed. 01963 01964 - shmem_context_size 01965 This sets an upper limit on the amount of shared memory for a single 01966 Veil shared memory context (there will be two of these). It defaults 01967 to 16K. Increase this if you have many shared memory structures. 01968 01969 \subsection Regression Regression Tests 01970 Veil comes with a built-in regression test suite. Use <code>make 01971 regress</code> or <code>make check</code> (after installing and 01972 configuring Veil) to run this. You will need superuser access to 01973 Postgres in order to create the regression test database. The 01974 regression test assumes you will have a postgres superuser account named 01975 the same as your OS account. If pg_hba.conf disallows "trust"ed access 01976 locally, then you will need to provide a password for this account in 01977 your .pgpass file (see postgres documentation for details). 01978 01979 The regression tests are all contained within the regress directory and 01980 are run by the regress.sh shell script. Use the -h option to get 01981 fairly detailed help. 01982 01983 \subsection Debugging Debugging 01984 If you encounter problems with Veil, you may want to try building with 01985 debug enabled. Define the variable VEIL_DEBUG on the make command line 01986 to add extra debug code to the executable: 01987 \verbatim 01988 $ make clean; make VEIL_DEBUG=1 all 01989 \endverbatim 01990 01991 This is a transient feature and not as pervasive as it could be. If you 01992 need help with debugging please contact the author. 01993 01994 Next: \ref History 01995 01996 */ 01997 /*! \page History History and Compatibility 01998 \section past Changes History 01999 \subsection v1_0 Version 9.1.0 (Stable) (2011-07-22) 02000 This is the first version of Veil to be considered production ready and 02001 completely stable. It is for use only with PostgreSQL 9.1. Support for 02002 older versions of PostgreSQL has been removed in this version. 02003 02004 Major changes include: 02005 - revamp of the build system to use PGXS and the new PostgreSQL 9.1 02006 extensions mechanism. Veil is now built as an extension. 02007 - modification to the veil_init() mechanism, allowing custom 02008 initialisation functions to be registered through the table 02009 veil.veil_init_fns 02010 - removal of the old veil_trial mechanism, which allowed Veil to be 02011 tried out without fully installing it. This has removed much 02012 unnecessary complexity. 02013 - much general code cleanup, including the removal of conditional code 02014 for older PostgreSQL versions. 02015 - documentation changes, including improved comments for Veil 02016 functions. 02017 02018 \subsection v9_12 Version 0.9.12 (2010-11-19) 02019 Release for compatibility with PostgreSQL V9.0. Minor bugfixes and 02020 improvements to the build system. Also added documentation about Veil's 02021 limitations. 02022 02023 \subsection v9_11 Version 0.9.11 (2010-03-12) 02024 Bugfix release, fixing a serious memory corruption bug that has existed 02025 in all previous versions. Users are strongly encouraged to avoid using 02026 older versions of Veil. 02027 02028 The version number has been deliberatley bumped past 0.9.10 to emphasize 02029 that the last part of the version is a two digit number. 02030 02031 \subsection v9_9 Version 0.9.9 (2009-07-06) 02032 New release to coincide with PostgreSQL V8.4. 02033 02034 \subsection v9_8 Version 0.9.8 (2008-02-06) 02035 This is the first Beta release. It incorporates a few bug fixes, a new 02036 serialisation API, improvements to the autoconf setup and makefiles, and 02037 some documentation improvements. The status of Veil has been raised to 02038 Beta in recognition of its relative stability. 02039 02040 \subsection v9_6 Version 0.9.6 (2008-02-06) 02041 This release has minor changes to support PostgreSQL 8.3. 02042 02043 \subsection v9_5 Version 0.9.5 (2007-07-31) 02044 This is a bugifx release, fixing a memory allocation bug in the use of 02045 bitmap_refs. There are also fixes for minor typos, etc. 02046 02047 \subsection v9_4 Version 0.9.4 (2007-02-21) 02048 This is a bugifx release, providing: 02049 - fix for major bug with recursive handling of spi connect, etc; 02050 - improvement to session initialisation code to do more up-front work 02051 in ensure_init(); 02052 - safer initialisation of malloc'd data structures; 02053 - improved error messages for shared memory exhaustion cases; 02054 - addition of debug code including canaries in data structures; 02055 - improvement to autoconf to better support Debian GNU/Linux, and OSX; 02056 - improvement to autoconf/make for handling paths containing spaces; 02057 - improvement to regression tests to better support OSX; 02058 - removal of spurious debug warning messages. 02059 02060 \subsection v9_3 Version 0.9.3 (2006-10-31) 02061 This version uses the new Postgres API for reserving shared memory for 02062 add-ins. It also allows the number of Veil-enabled databases for a 02063 cluster to be configured, and refactors much of the shared memory code. 02064 A small fix for the Darwin makefile was also made. 02065 02066 \subsection v9_2 Version 0.9.2 (2006-10-01) 02067 This version was released to coincide with Postgres 8.2beta1 and first 02068 made use of new Postgres APIs to allow Veil to be a good Postgres 02069 citizen. 02070 02071 With prior versions of Veil, or prior versions of Postgres, Veil steals 02072 from Postgres the shared memory that it requires. This can lead to the 02073 exhaustion of Postgres shared memory. 02074 02075 Unfortunately, the Postgres API for shared memory reservation had to 02076 change follwing 8.2.beta1, and this version of Veil is therefore deprecated. 02077 02078 \subsection v9_1 Version 0.9.1 (2006-07-04) 02079 This release fixed a small number of bugs and deficiencies: 02080 - major error in veil_perform_reset that prevented proper use of the two 02081 interdependant shared memory contexts 02082 - minor improvements in the build process to "configure" and friends 02083 - minor documentation improvements 02084 02085 \subsection v9_0 Version 0.9.0 (2005-10-04) 02086 This was the first public alpha release of Veil. 02087 02088 \section forecast Change Forecast 02089 New versions will be released with each new major version of 02090 PostgreSQL. Once there are three PostgreSQL versions for which Veil has 02091 been at production status, the change history and support matrix for for 02092 pre-production versions will be removed from this documentation. 02093 02094 \section compatibility Supported versions of Postgres 02095 <TABLE> 02096 <TR> 02097 <TD rowspan=2>Veil version</TD> 02098 <TD colspan=9>Postgres Version</TD> 02099 </TR> 02100 <TR> 02101 <TD>7.4</TD> 02102 <TD>8.0</TD> 02103 <TD>8.1</TD> 02104 <TD>8.2beta1</TD> 02105 <TD>8.2</TD> 02106 <TD>8.3</TD> 02107 <TD>8.4</TD> 02108 <TD>9.0</TD> 02109 <TD>9.1</TD> 02110 </TR> 02111 <TR> 02112 <TD>0.9.0 Alpha</TD> 02113 <TD>1</TD> 02114 <TD>1</TD> 02115 <TD>1</TD> 02116 <TD>-</TD> 02117 <TD>-</TD> 02118 <TD>-</TD> 02119 <TD>-</TD> 02120 <TD>-</TD> 02121 <TD>-</TD> 02122 </TR> 02123 <TR> 02124 <TD>0.9.1 Alpha</TD> 02125 <TD>1</TD> 02126 <TD>1</TD> 02127 <TD>1</TD> 02128 <TD>-</TD> 02129 <TD>-</TD> 02130 <TD>-</TD> 02131 <TD>-</TD> 02132 <TD>-</TD> 02133 <TD>-</TD> 02134 </TR> 02135 <TR> 02136 <TD>0.9.2 Alpha</TD> 02137 <TD>-</TD> 02138 <TD>1</TD> 02139 <TD>1</TD> 02140 <TD>2</TD> 02141 <TD>-</TD> 02142 <TD>-</TD> 02143 <TD>-</TD> 02144 <TD>-</TD> 02145 <TD>-</TD> 02146 </TR> 02147 <TR> 02148 <TD>0.9.3 Alpha</TD> 02149 <TD>-</TD> 02150 <TD>1</TD> 02151 <TD>1</TD> 02152 <TD>-</TD> 02153 <TD>3</TD> 02154 <TD>-</TD> 02155 <TD>-</TD> 02156 <TD>-</TD> 02157 <TD>-</TD> 02158 </TR> 02159 <TR> 02160 <TD>0.9.4 Alpha</TD> 02161 <TD>-</TD> 02162 <TD>1</TD> 02163 <TD>1</TD> 02164 <TD>-</TD> 02165 <TD>3</TD> 02166 <TD>-</TD> 02167 <TD>-</TD> 02168 <TD>-</TD> 02169 <TD>-</TD> 02170 </TR> 02171 <TR> 02172 <TD>0.9.5 Alpha</TD> 02173 <TD>-</TD> 02174 <TD>1</TD> 02175 <TD>1</TD> 02176 <TD>-</TD> 02177 <TD>3</TD> 02178 <TD>-</TD> 02179 <TD>-</TD> 02180 <TD>-</TD> 02181 <TD>-</TD> 02182 </TR> 02183 <TR> 02184 <TD>0.9.6 Alpha</TD> 02185 <TD>-</TD> 02186 <TD>1</TD> 02187 <TD>1</TD> 02188 <TD>-</TD> 02189 <TD>3</TD> 02190 <TD>3</TD> 02191 <TD>-</TD> 02192 <TD>-</TD> 02193 <TD>-</TD> 02194 </TR> 02195 <TR> 02196 <TD>0.9.8 Beta</TD> 02197 <TD>-</TD> 02198 <TD>1</TD> 02199 <TD>1</TD> 02200 <TD>-</TD> 02201 <TD>3</TD> 02202 <TD>3</TD> 02203 <TD>-</TD> 02204 <TD>-</TD> 02205 <TD>-</TD> 02206 </TR> 02207 <TR> 02208 <TD>0.9.9 Beta</TD> 02209 <TD>-</TD> 02210 <TD>1</TD> 02211 <TD>1</TD> 02212 <TD>-</TD> 02213 <TD>3</TD> 02214 <TD>3</TD> 02215 <TD>3</TD> 02216 <TD>-</TD> 02217 <TD>-</TD> 02218 </TR> 02219 <TR> 02220 <TD>0.9.11 Beta</TD> 02221 <TD>-</TD> 02222 <TD>1</TD> 02223 <TD>1</TD> 02224 <TD>-</TD> 02225 <TD>3</TD> 02226 <TD>3</TD> 02227 <TD>3</TD> 02228 <TD>-</TD> 02229 <TD>-</TD> 02230 </TR> 02231 <TR> 02232 <TD>0.9.12 Beta</TD> 02233 <TD>-</TD> 02234 <TD>-</TD> 02235 <TD>-</TD> 02236 <TD>-</TD> 02237 <TD>-</TD> 02238 <TD>3</TD> 02239 <TD>3</TD> 02240 <TD>3</TD> 02241 <TD>-</TD> 02242 </TR> 02243 <TR> 02244 <TD>9.1.0 (Stable)</TD> 02245 <TD>-</TD> 02246 <TD>-</TD> 02247 <TD>-</TD> 02248 <TD>-</TD> 02249 <TD>-</TD> 02250 <TD>-</TD> 02251 <TD>-</TD> 02252 <TD>-</TD> 02253 <TD>Yes</TD> 02254 </TR> 02255 </TABLE> 02256 Notes: 02257 02258 1) These combinations of Veil and Postgres provide no configuration 02259 options for shared memory. Veil's shared memory may be exhausted by 02260 too many requests for large shared objects. Furthermore, Postgres' 02261 own shared memory may be easily exhausted by creating too many 02262 Veil-using databases within a cluster. 02263 02264 2) This version is deprecated 02265 02266 3) These combinations of Veil and Postgres provide full configuration 02267 options for shared memory usage, and Veil cooperates with Postgres 02268 for the allocation of such memory meaning that it is not possible to 02269 use Veil to exhaust Postgres' shared memory. This is the minimum 02270 Veil configuration recommended for production use. 02271 02272 \section platforms Supported Platforms 02273 Veil should be buildable on any platform supported by PostgreSQL and 02274 PGXS. 02275 02276 Next: \ref Feedback 02277 02278 */ 02279 /*! \page Feedback Bugs and Feedback 02280 \section Feedback Bugs and Feedback 02281 For general feedback, to start and follow discussions, etc please join 02282 the veil-general@pgfoundry.org mailing list. 02283 02284 If you wish to report a bug or request a feature, please send mail to 02285 veil-general@pgfoundry.org 02286 02287 If you encounter a reproducible veil bug that causes a database server 02288 crash, a gdb backtrace would be much appreciated. To generate a 02289 backtrace, you will need to login to the postgres owner account on the 02290 database server. Then identify the postgres backend process associated 02291 with the database session that is going to crash. The following command 02292 identifies the backend pid for a database session started by marc: 02293 02294 \verbatim 02295 $ ps auwwx | grep ^postgres.*ma[r]c | awk '{print $2}' 02296 \endverbatim 02297 02298 Now you invoke gdb with the path to the postgres binary and the pid for 02299 the backend, eg: 02300 02301 \verbatim 02302 $ gdb /usr/lib/postgresql/9.1/bin/postgres 5444 02303 \endverbatim 02304 02305 Hit c and Enter to get gdb to allow the session to continue. Now, 02306 reproduce the crash from your database session. When the crash occurs, 02307 your gdb session will return to you. Now type bt and Enter to get a 02308 backtrace. 02309 02310 If you wish to contact the author offlist, you can find him at his website 02311 http://bloodnok.com/Marc.Munro 02312 02313 Next: \ref Performance 02314 02315 */ 02316 /*! \page Performance Performance 02317 \section perf Performance 02318 Attempts to benchmark veil using pgbench have not been very successful. 02319 It seems that the overhead of veil is small enough that it is 02320 overshadowed by the level of "noise" within pgbench. 02321 02322 Based on this inability to properly benchmark veil, the author is going 02323 to claim that "it performs well". 02324 02325 To put this into perspective, if your access functions do not require 02326 extra fetches to be performed in order to establish your access rights, 02327 you are unlikely to notice or be able to measure any performance hit 02328 from veil. 02329 02330 If anyone can provide good statistical evidence of a performance hit, 02331 the author would be most pleased to hear from you. 02332 02333 Next: \ref Credits 02334 02335 */ 02336 /*! \page Credits Credits 02337 \section Credits 02338 The Entity Relationship Diagram in section \ref demo-erd was produced 02339 automatically from an XML definition of the demo tables, using Autograph 02340 from CF Consulting. Thanks to Colin Fox for allowing its use. 02341 02342 Thanks to the PostgreSQL core team for providing PostgreSQL. 02343 02344 Thanks to pgfoundry for providing a home for this project. 02345 */ 02346