[python]如何自己跟踪训练过程中的损失值并可视化出来

发布于2019-09-11 11:54 阅读(1311) 评论(0) 点赞(5) 收藏(1)

1 跟踪训练函数的损失值

如果跑的代码中没有显示损失函数的变化趋势，而自己需要根据这个来调整超参，那么可以自己编写函数去实现这一个需求。
首先应该将每次的损失值记录并储存下来。这里以CornerNet的代码为例，代码链接如下：
CornerNet
在train.py函数中，损失值在如下的部分得到：

    with stdout_to_tqdm() as save_stdout:
        for iteration in tqdm(range(start_iter + 1, max_iteration + 1), file=save_stdout, ncols=80):
            training = pinned_training_queue.get(block=True)
            training_loss = nnet.train(**training)

            if display and iteration % display == 0:
                print("training loss at iteration {}: {}".format(iteration, training_loss.item()))
            del training_loss

            # if val_iter and validation_db.db_inds.size and iteration % val_iter == 0:
            #     nnet.eval_mode()
            #     validation = pinned_validation_queue.get(block=True)
            #     validation_loss = nnet.validate(**validation)
            #     print("validation loss at iteration {}: {}".format(iteration, validation_loss.item()))
            #     nnet.train_mode()

            if iteration % snapshot == 0:
                nnet.save_params(iteration)

            if iteration % stepsize == 0:
                learning_rate /= decay_rate
                nnet.set_lr(learning_rate)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

如果想要记录并储存好其值，加入以下代码即可：

    with stdout_to_tqdm() as save_stdout:
        for iteration in tqdm(range(start_iter + 1, max_iteration + 1), file=save_stdout, ncols=80):
            training = pinned_training_queue.get(block=True)
            training_loss = nnet.train(**training)
            loss = training_loss.cpu()
            loss_ = str(loss.data.numpy())
            with open('path', 'a') as f:
                f.write(str(iteration))
                f.write(' ')
                f.write(loss_)
                if iteration < max_iteration:
                    f.write(' \r\n')
            if display and iteration % display == 0:
                print("training loss at iteration {}: {}".format(iteration, training_loss.item()))
            del training_loss

            # if val_iter and validation_db.db_inds.size and iteration % val_iter == 0:
            #     nnet.eval_mode()
            #     validation = pinned_validation_queue.get(block=True)
            #     validation_loss = nnet.validate(**validation)
            #     print("validation loss at iteration {}: {}".format(iteration, validation_loss.item()))
            #     nnet.train_mode()

            if iteration % snapshot == 0:
                nnet.save_params(iteration)

            if iteration % stepsize == 0:
                learning_rate /= decay_rate
                nnet.set_lr(learning_rate)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

加入的部分为：

            loss = training_loss.cpu()
            loss_ = str(loss.data.numpy())
            with open('path', 'a') as f:
                f.write(str(iteration))
                f.write(' ')
                f.write(loss_)
                if iteration < max_iteration:
                    f.write(' \r\n')
1
2
3
4
5
6
7
8

解释一下代码
由于深度学习计算loss的时候基本上loss都是cuda的一个tensor变量，储存在cuda中的。是不能够直接复制过来的，所以需要用.cpu()把cuda中的值转移给cpu中储存
然后由于此时转移过去以后还是一个variable变量（就是可以backpropogation来计算grad的一种变量），所以需要用variable.data把其中的数据单独取出来，但是此时还是一个tensor，需要转换成numpy，所以再通过一个.numpy()
f.write（）中的必须是一个string类型的数据，所以要转成string
最后if语句是为了在最后一行的时候不要再写入回车了。
path:就是你想把记录下来的数据储存在哪个文件夹下，比如./loss.txt，就是储存在当前路径下的txt文件。这里建议保存成txt文件，以后要用的话就很方便处理。这个txt文件不需要自己新建一个空白，没有的话python会自己建立一个的
open函数中我选择用a参数，而不是用w，这是因为w会抹去之前写过的重新写，最后只剩下最后一对数据了，而我们的目的明显不是这样的，a是从上一次的位置接着写，这就nice了。

2 根据保存好的训练过程中的损失值可视化

"""



Note: The code is used to show the change trende via the whole training procession.
First: You need to mark all the loss of every iteration
Second: You need to write these data into a txt file with the format like:
......
iter loss
iter loss
......
Third: the path is the txt file path of your loss



"""

import matplotlib.pyplot as plt


def read_txt(path):
    with open(path, 'r') as f:
        lines = f.readlines()
    splitlines = [x.strip().split(' ') for x in lines]
    return splitlines


# Referenced from Tensorboard(a smooth_loss function:https://blog.csdn.net/charel_chen/article/details/80364841)
def smooth_loss(path, weight=0.85):
    iter = []
    loss = []
    data = read_txt(path)
    for value in data:
        iter.append(int(value[0]))
        loss.append(int(float(value[1])))
        # Note a str like '3.552' can not be changed to int type directly
        # You need to change it to float first, can then you can change the float type ton int type
    last = loss[0]
    smoothed = []
    for point in loss:
        smoothed_val = last * weight + (1 - weight) * point
        smoothed.append(smoothed_val)
        last = smoothed_val
    return iter, smoothed


if __name__ == "__main__":
    path = './loss.txt'
    loss = []
    iter = []
    iter, loss = smooth_loss(path)
    plt.plot(iter, loss, linewidth=2)
    plt.title("Loss-iters", fontsize=24)
    plt.xlabel("iters", fontsize=14)
    plt.ylabel("loss", fontsize=14)
    plt.tick_params(axis='both', labelsize=14)
    plt.savefig('./loss_func.png')
    plt.show()
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58