This is mainly framed towards Numpy arrays but I feel it's a general design problem. In many cases I have the following generic problem in scientific computing: I have to read a dataset composed of several time instances of a certain vector field, e.g. the time-varying velocity field on a number of points. Something that could be a 3D array of sizes (Npoints, 3, Ntimesteps) (3 because I have three components of a vector). I normally would have one file per timestep, so I'd have to read Ntimesteps files, but I don't know beforehand the size of each field (i.e. I don't know Npoints beforehand, but it's the same for each file). If I want to preallocate the Numpy array to store this data, I got used to doing something along these lines:

tsteps = list(glob.glob('time*.csv'))
Nsteps = len(tsteps)
with open(tsteps[0], 'r') as f:
    # do my work to get Npoints and the data of the first file
    dataset = np.zeros(Npoints, 3, Nsteps)
    dataset[:,:,0] = data_first_file
for i,f in enumerate(tsteps[1:]):
    # do my work again
    dataset[:,:,i] = data_from_tstep_i

However, this does not look very 'elegant' since I have to write twice the code to process each timestep file: once to preallocate the array and once in the loop. Is there a more elegant pattern to do this?

解决方案

i think that there's no elegant way to do it if you don't know the shape in advance. but you should initiate it inside the loop for readability

tsteps = glob.glob('time*.csv') 
for i, filename in enumerate(tsteps):
    with open(filename) as f:
        data_i = np.loadtxt(f)
        if i==0:
            dataset = np.zeros(data_i.shape[0], 3, len(tsteps))
        dataset[:,:,i] = data_i