Using org-entries like a database
Posted August 21, 2014 at 08:45 PM | categories: orgmode, dft | tags: | View Comments
Updated August 22, 2014 at 10:27 AM
Table of Contents
- 1. Generating the entries
- 1.1. 10/eos-exp/f+0 10 eos_dash_exp f_plus_0
- 1.2. 10/eos-exp/f+10 10 eos_dash_exp f_plus_10
- 1.3. 10/eos-exp/f+12 10 eos_dash_exp f_plus_12
- 1.4. 10/eos-exp/f+15 10 eos_dash_exp f_plus_15
- 1.5. 10/eos-exp/f+2 10 eos_dash_exp f_plus_2
- 1.6. 10/eos-exp/f+4 10 eos_dash_exp f_plus_4
- 1.7. 10/eos-exp/f+6 10 eos_dash_exp f_plus_6
- 1.8. 10/eos-exp/f+8 10 eos_dash_exp f_plus_8
- 1.9. 10/eos-exp/f-10 10 eos_dash_exp f_dash_10
- 1.10. 10/eos-exp/f-12 10 eos_dash_exp f_dash_12
- 1.11. 10/eos-exp/f-15 10 eos_dash_exp f_dash_15
- 1.12. 10/eos-exp/f-2 10 eos_dash_exp f_dash_2
- 1.13. 10/eos-exp/f-4 10 eos_dash_exp f_dash_4
- 1.14. 10/eos-exp/f-6 10 eos_dash_exp f_dash_6
- 1.15. 10/eos-exp/f-8 10 eos_dash_exp f_dash_8
- 1.16. 11/eos-exp/f+0 11 eos_dash_exp f_plus_0
- 1.17. 11/eos-exp/f+10 11 eos_dash_exp f_plus_10
- 1.18. 11/eos-exp/f+12 11 eos_dash_exp f_plus_12
- 1.19. 11/eos-exp/f+15 11 eos_dash_exp f_plus_15
- 1.20. 11/eos-exp/f+2 11 eos_dash_exp f_plus_2
- 1.21. 11/eos-exp/f+4 11 eos_dash_exp f_plus_4
- 1.22. 11/eos-exp/f+6 11 eos_dash_exp f_plus_6
- 1.23. 11/eos-exp/f+8 11 eos_dash_exp f_plus_8
- 1.24. 11/eos-exp/f-10 11 eos_dash_exp f_dash_10
- 1.25. 11/eos-exp/f-12 11 eos_dash_exp f_dash_12
- 1.26. 11/eos-exp/f-15 11 eos_dash_exp f_dash_15
- 1.27. 11/eos-exp/f-2 11 eos_dash_exp f_dash_2
- 1.28. 11/eos-exp/f-4 11 eos_dash_exp f_dash_4
- 1.29. 11/eos-exp/f-6 11 eos_dash_exp f_dash_6
- 1.30. 11/eos-exp/f-8 11 eos_dash_exp f_dash_8
- 2. Tabulating the results in column mode
- 3. org-map-entries to get the data
- 4. Getting specific data from a calculation
- 5. Summary
I recently added an org export to jasp which provides an org-mode representation of a Vasp calculation. It is still a work in progress, but the main ideas are developed. The idea is to represent a calculation as an org-mode headline with tabular data, properties and tags. This will make the representation human readable, but machine accessible. Here I examine how we could use it.
1 Generating the entries
We will loop over a set of calculations that define the equation of state for two different Al-Ni alloy structures. For each calculation this code generates a level 3 headline with some tags derived from the calculation directory. I chose level 3 so they would all be nested in this level 2 headline. We write out two sets of data because later we will selectively get portions of the data.
from jasp import * from jasp.utils import get_jasp_dirs for d in get_jasp_dirs('10'): with jasp(d) as calc: print calc.org(level=3) for d in get_jasp_dirs('11'): with jasp(d) as calc: print calc.org(level=3)
2 Tabulating the results in column mode
We can use column view to see the results in a tabular like form. For example, here we capture the VOLUME and TOTALENERGY properties.
That is more or less what we want, but we cannot use that table as a data source very directly. While you can use it as a data source, you just get a string of the whole table, e.g.
print data.__class__
<type 'str'>
You could parse that string, or try converting it to a table, but we skip that for now. In the next section we consider another approach that more directly gets at the data.
3 org-map-entries to get the data
Let us try a different approach. Here we map over headlines that match a criteria, and get specific data. We add a header of labels and horizontal line to the beginning of the table. Here we find the results tagged with 11, and that match fdash* (we actually use a regular expression).
(cons '(label volume energy) (cons 'hline (org-map-entries (lambda () (list (nth 4 (org-heading-components)) (org-entry-get (point) "VOLUME") (org-entry-get (point) "TOTAL_ENERGY"))) ;; this next line specifies headlines tagged with 11 ;; and that match the regular expression in {} "+11+{f_dash_.*}")))
Now, since there are strings in the table (or in the results from the named code-block), we will have to cast the results to floats using a numpy array. Note it appears that we actually run the code block to get the results, they do not come from the table. The table is just helpful to know what data you are using.
import matplotlib.pyplot as plt import numpy as np print E plt.plot(np.array(V), np.array(E), 'ko') plt.xlabel('Volume $\AA^3$') plt.ylabel('Energy (eV)') plt.savefig('11-eos.png')
['-20.038179', '-19.912894', '-19.662755', '-20.275411', '-20.249632', '-20.202731', '-20.133749']
And here you have the figure. We extracted some data into an org-file in human readable form. Then, we extracted the data from those entries to a table, and finally plotted it.
4 Getting specific data from a calculation
Say you want the stress from some calculation. Each heading is identified by a CUSTOMID, so we can temporarily go to that headline, and get the property. Easy!
(save-excursion (org-open-link-from-string "[[#10/eos-exp/f+0]]") (org-entry-get (point) "STRESS"))
[-0.003191325893994518, -0.003191325893994518, -0.003191325893994518, -0.00042576435655845916, 0.00042576435655845916, 0.00042576435655845916]
Note, however, that the value returned is a string, so we need to convert it to numbers if we want to do anything with it, e.g. find the maximum component. Here, we eval the string in Python to make it a list, and then find the maximum value.
import numpy as np print np.abs(np.array(eval(stress))).max()
0.00319132589399
Next, we get the unit cell from that calculation, so we can calculate the volume using python. The unit cell is stored in a table and we extract the unit cell vectors from it in the code block header.
import numpy as np print('cell = {}'.format(np.array(data))) print('Volume = {}'.format(np.linalg.det(np.array(data))))
cell = [[-1.98408 0. -1.98408] [ 0. -1.98408 1.98408] [-3.53625 3.53625 3.53625]] Volume = 41.7621235495
Compare that to the volume we computed when the entry was generated, and you can see they are the same.
(save-excursion (org-open-link-from-string "[[#10/eos-exp/f+0]]") (org-entry-get (point) "VOLUME"))
41.7621235495
The last example is getting the formula for the calculation. The atoms positions are stored in a table in the entry, and the species for each atom is stored in the first column. We extract the species into a variable, then then get counts for each species, and finally construct a string representation of the formula.
(let* ((unique-species '()) (counts)) ;; first get unique species (mapcar (lambda (s) (add-to-list 'unique-species s)) species) ;; now get counts (setq counts (mapcar (lambda (s) (--count (string= it s) species)) unique-species)) ;; Now put them together (mapconcat 'identity (cl-mapcar (lambda (s n) (format "%s%s" s n)) unique-species counts) ""))
Ni1Al2
Suppose we want to sort the elements alphabetically. We can make a list of cons cells of element and count, sort the list by element, and then put them back together. We do this in lisp.
(let* ((unique-species '()) (counts)) ;; first get unique species (mapcar (lambda (s) (add-to-list 'unique-species s)) species) ;; now get counts (setq counts (mapcar (lambda (s) (--count (string= it s) species)) unique-species)) (mapconcat (lambda (x) (format "%s%s" (car x) (cdr x))) (cl-sort (cl-mapcar (lambda (a b) (cons a b)) unique-species counts) 'string-lessp :key 'car) ""))
Al2Ni1
That is not too bad. It is a little verbose, some of that could be put into a library to be reused. For example, here is the same code in Python for the unsorted formula
from collections import Counter counts = Counter(species) print ''.join(['{0}{1}'.format(x, counts[x]) for x in counts])
Ni1Al2
and for the sorted formula.
from collections import Counter counts = Counter(species) print ''.join(['{0}{1}'.format(x, counts[x]) for x in sorted(counts.keys())])
Al2Ni1
Obviously a lot must be happening behind the scenes in those short, dense python blocks. For example, the list comprehension syntax in Python is functionally similar to the mapcar code, and the join function is similar to the mapconcat code in lisp.
5 Summary
This has some potential. It is a hybrid approach to human and machine readable data, that enables inline analysis. You can mix programming languages, which is great if one language has unique capabilities or is just more convenient. There are some limitations, mostly that org-properties are read as strings, and they have to be converted to the right data type, e.g. numbers or lists or arrays, etc… But, these can be saved as forms that can be directly evaled in some languages. On the other hand, it is possible to do analysis on another machine where the original data does not exist.
This is a very different mental model of data than I am used to. We typically have thought about data in files, and paths to get to that data. Here, data is in headings, and we use links to get to the data. We also use different syntax to link data to code blocks. That means in some cases the code blocks cannot be run independently of this file.
Copyright (C) 2014 by John Kitchin. See the License for information about copying.
Org-mode version = 8.2.7c