snapshotState 和 notifyCheckpointComplete 时机
先snapshotState, 为了关闭文件. 文件在关闭之前都是空的.
private InProgressFileWriter.PendingFileRecoverable closePartFile() throws IOException {
InProgressFileWriter.PendingFileRecoverable pendingFileRecoverable = null;
if (inProgressPart != null) {
// 关闭文件
pendingFileRecoverable = inProgressPart.closeForCommit();
pendingFileRecoverablesForCurrentCheckpoint.add(pendingFileRecoverable);
inProgressPart = null;
}
return pendingFileRecoverable;
}普通的一个数据到来时候:
从invoke走起.
若触发滚动条件org.apache.flink.streaming.api.functions.sink.filesystem.Bucket#rollPartFile
否则添加到writer中
我们用的是org.apache.flink.formats.compress.writers.HadoopCompressionBulkWriter
