edited documentation about load_or_create

This commit is contained in:
JJJHolscher 2024-11-12 14:11:16 +01:00
parent 7048bb058a
commit 90146757f1

123
README.md
View File

@ -5,39 +5,54 @@ It might get its own package someday, when it's less before-pre-alpha-sneakpeak-
## load_or_create ## load_or_create
The main idea is that an object's storage path should be inferable from the arguments with which it was created.
In reality we often keep track of an object and its path seperately, which can work out poorly.
The `load_or_create` function bundles an object's creation, load, save and path function together such that anyone working with the code less often has to think about storage locations and more often can focus on the functionality of their code.
### the status quo ### the status quo
A common pattern is: A common pattern has you write the following 4 functions for objects that are expensive to create:
```python ```python
def load_obj(path): from pathlib import Path
"" import pickle # We use pickle as an example here.
def create_obj(name, some_kwarg=1):
"""This function takes long, so we don't want to run it redundantly."""
return obj
def save_obj(obj, path): def save_obj(obj, path):
"" """Instead we write a save function..."""
with open(path, "wb") as file:
pickle.dump(obj, file)
def create_obj(*args, **kwargs): def load_obj(path):
"" """...and a load function so we only need to create the object once."""
with open(path, "rb") as file:
return pickle.load(file)
def infer_path(obj_id): def infer_path(name):
return "./obj/" + str(obj_id) """We need to keep track where the created object is stored."""
return "./obj/" + str(name) + ".pkl"
obj_id = 0 # When you have the above 4 fuctions, this pattern will start to occur.
path = infer_path(obj_id) name = "MyObject"
if path.exists(): path = infer_path(name)
if Path(path).exists():
obj = load_obj(load) obj = load_obj(load)
else: else:
obj = create_obj(obj_id, some_kwarg=0) obj = create_obj(name, some_kwarg=0)
save_obj(obj, path) save_obj(obj, path)
``` ```
And in some cases, you want to create and save many variations of the object. In some cases, you want to create and save many variations of the object.
It might be better to hash it's characteristics and use that as part of the path. It might be better to hash its characteristics and use that as part of the path.
```python ```python
import sha256 import sha256
import json import json
def infer_path(obj_id, **some_other_kwargs): def infer_path(name, **some_other_kwargs):
hash = str(sha256(json.dumps(some_other_kwargs)).hexdigest()) hash = str(sha256(json.dumps(some_other_kwargs)).hexdigest())
return "./obj/" + hash + ".pkl" return "./obj/" + hash + ".pkl"
``` ```
@ -45,21 +60,21 @@ def infer_path(obj_id, **some_other_kwargs):
### the problem ### the problem
The above is fine and dandy, but when someone wants to use your obj, The above is fine and dandy, but when someone wants to use your obj,
they'd need to keep track of 4 separate functions. they'd need to keep track of your 4 separate functions.
You can dress it up as such: You can dress it up as such:
```python ```python
def get_obj(obj_id, *args, **kwargs): def get_obj(name, some_kwarg):
path = infer_path(obj_id) path = infer_path(name)
if path.exists(): if path.exists():
obj = load_obj(load) obj = load_obj(load)
else: else:
obj = create_obj(obj_id, some_kwarg=0) obj = create_obj(name, some_kwarg=some_kwarg)
save_obj(obj, path) save_obj(obj, path)
return obj return obj
``` ```
But that takes a lot of freedom away from your user, who might have their But that takes a lot of freedom away from your user, who might have their
own ideas on where the object should be stored. own ideas on where and how the object should be loaded or stored.
### the solution ### the solution
@ -71,11 +86,14 @@ get_obj = load_or_create(
path_fn=infer_path, path_fn=infer_path,
)(create_obj) )(create_obj)
obj = get_obj(obj_id, some_kwarg=0) obj = get_obj(name, some_kwarg=0)
path_of_obj_0 = get_obj.path(obj_id, some_kwarg=0) # We can now infer the path of an object from its creation arguments.
path_of_obj_1 = get_obj.path_of_obj(obj) path = get_obj.path(name, some_kwarg=0)
assert path_of_obj_0 == path_of_obj_1
# We also can use `get_obj.path_of_obj` to recall the path of any object
# that `get_obj` returned in he past.
assert path == get_obj.path_of_obj(obj)
``` ```
You can now elegantly pack the four functions together. You can now elegantly pack the four functions together.
@ -88,14 +106,20 @@ get_obj.path_fn = lambda hash: f"./{hash}.pkl"
Now, storing different objects of which one is dependent on the other, becomes intuitive and elegant: Now, storing different objects of which one is dependent on the other, becomes intuitive and elegant:
```python ```python
# This code is written at the library level
get_human = load_or_create( get_human = load_or_create(
path_fn=lambda name: "./" + name + "/body.txt" path_fn=lambda name: "./" + name + "/body.pkl"
# If you omit the save and load functions, load_or_create will use pickle.
)(lambda name: name) )(lambda name: name)
get_finger_print = load_or_create( get_finger_print = load_or_create(
path_fn=lambda finger: get_human.dir_from_obj(human) / f"{finger}.print" path_fn=lambda human, finger: get_human.dir_from_obj(human) / f"{finger}.print"
)(lambda human, finger: f"{human}'s finger the {finger}") )(lambda human, finger: f"{human}'s finger the {finger}")
assert not get_human.path("john").exists() # This code is what a user can work with.
assert not get_human.path("john").exists() # ./john/body.pkl
human = get_human("john") human = get_human("john")
assert get_human.path("john").exists() assert get_human.path("john").exists()
@ -103,11 +127,10 @@ finger_print = get_finger_print(human, "thumb")
assert get_finger_print.path(human, "thumb") == "./john/thumb.print" assert get_finger_print.path(human, "thumb") == "./john/thumb.print"
``` ```
The Finger print is now always stored in the same directory as where the human's `body.txt` is stored. The finger print is now always stored in the same directory as where the human's `body.pkl` is stored.
You don't need to keep track of the location of `body.txt`. You don't need to keep track of the location of `body.pkl`.
### four functions in one
### under the hood
The main trick is to match the parameter names of the `create` function (in our case `create_obj`) The main trick is to match the parameter names of the `create` function (in our case `create_obj`)
with those of the three other subfunctions (in our case `load_obj`, `save_obj` and `infer_path`). with those of the three other subfunctions (in our case `load_obj`, `save_obj` and `infer_path`).
@ -115,17 +138,31 @@ with those of the three other subfunctions (in our case `load_obj`, `save_obj` a
The three subfunctions's allowed parameters are mostly a non-strict superset of the create function's The three subfunctions's allowed parameters are mostly a non-strict superset of the create function's
parameters. parameters.
When you call `get_obj`, something like this happens: When you call the `load_or_create`-wrapped `get_obj`, something like this happens:
```python ```python
def call_fn_with_filtered_arguments(fn, *args, **kwargs): def call_fn_with_filtered_arguments(fn, *args, **kwargs):
""" call fn with only the subset of args and kwargs that fn expects. """ call `fn` with only the subset of `args` and `kwargs` that it expects.
This is necessary, as python will complain if a function receives any
argument for which there is no function parameter.
So
def fn(a):
pass
fn(a=0, b=1)
will error, so we need to remove b before calling fn.
This example function is wrong, if you're curious you need to check the
source code.
""" """
path_parameters = get_parameters_that_fn_expects(infer_path) # Get the names of the paremeters that `fn` accepts.
# in reality we first infer the args name, for positional arguments. path_parameters = get_parameters_that_fn_expects(fn)
args = [arg for arg in args if arg in path_parameters] # Filter for positinoal arguments that `fn` accepts.
kwargs = {key: arg for key, arg in kwargs.items() if key in path_parameters} args = [a for i, a in enumerate(args) if name_of_positional(i, fn) in path_parameters]
return infer_path(*args, **kwargs) # Filter for keyword arguments that `fn` accepts.
kwargs = {k: a for k, a in kwargs.items() if k in path_parameters}
# Call `fn` with the filtered subset of the original args and kwargs.
return fn(*args, **kwargs)
def get_obj_pseudo_code(*args, **kwargs): def get_obj_pseudo_code(*args, **kwargs):
hash = some_hash_fn(*args, **kwargs) hash = some_hash_fn(*args, **kwargs)
@ -162,14 +199,14 @@ during its creation call.
In reality, we tend to separately keep track of some object's path, its arguments and itself. In reality, we tend to separately keep track of some object's path, its arguments and itself.
This tends to go bad when we need to load, save or create the object in some other context. This tends to go bad when we need to load, save or create the object in some other context.
It becomes easy to forget where some object ought to be stored. It becomes easy to forget where some object ought to be stored.
Or it can happen that or different places where the same object is handled, have different opinions on storage location. Or it can happen that different places where the same object is handled, have different opinions on its storage location.
It can lead to duplicates; forgetting where the object was stored; or losing a folder of data It can lead to duplicates; forgetting where the object was stored; or losing a folder of data
because the folder is too unwieldy to salvage. because the folder is too unwieldy to salvage.
By packaging a function with it's load and save countparts and a default storage location, we don't By packaging a function with its load and save countparts and a default storage location, we don't
need to worry about storage location anymore and can focus on creating and using our objects. need to worry about the storage location anymore and can focus on creating and using our objects.
If we ever do change our minds on the ideal storage location, then there is an obvious central place If we ever do change our minds on the ideal storage location, then there is an obvious central place
where we can change it, and that change then easily immediately applies to _all_ the places where for changing it, and that change then easily immediately applies to _all_ the places where
some object's path needs to be determined. that object's path needs to be determined.