网页快照实现-Java探险之旅

摘要：引言在监控系统网页爬虫审核流程等领域，经常需要保存系统在某一时刻的状态网页快照。之后使用另一个命令行访问本地的端口能够看到调试信息应该就是安装成功了下载提供了操作的，是控制的桥梁。

在监控系统、网页爬虫、审核流程等领域，经常需要保存系统在某一时刻的状态-网页快照。实现网页快照主要有phantomjs、selenium,但是网上资料都是零碎的，本文主要采用Selenium,详细阐述从编码到部署的整个流程.

    org.seleniumhq.selenium
    selenium-java
    3.11.0

(1) 创建WebDriver

a.手机设备

public WebDriver createWebDriver(String device) {
    Map mobileEmulation = new HashMap();
    //设置设备,例如:Google Nexus 7/Apple iPhone 6
    //mobileEmulation.put("deviceName", "Google Nexus 7");
    if(StringUtils.isBlank(device))
        device = "iPhone 6";
    mobileEmulation.put("deviceName", device);   //这里是要使用的模拟器名称，就是浏览器中模拟器中的顶部型号
    Map chromeOptions = new HashMap();
    chromeOptions.put("mobileEmulation", mobileEmulation);
    DesiredCapabilities capabilities = DesiredCapabilities.chrome();
    capabilities.setCapability(ChromeOptions.CAPABILITY, chromeOptions);
    WebDriver driver = new ChromeDriver(capabilities);
    Dimension dimension = new Dimension(420, 900);
    driver.manage().window().setSize(dimension);
    return driver;
}

b.PC设备

public WebDriver createWebDriver(String device) {
    //暂时采用特定分辨率
    ChromeOptions options = new ChromeOptions();
    options.addArguments("window-size=1200,800");
    WebDriver driver = new ChromeDriver(options);
    driver.manage().window();
    return driver;
}

(2) 绘制图像

    public String execute(String uri, String device) {
        PerfCounter.count("miui_ad_schedule_capture.execute.total", 1L);
        long start = System.currentTimeMillis();
        System.setProperty("webdriver.chrome.driver", "/home/rd/chromedriver");
        //windows: System.setProperty("webdriver.chrome.driver", "src/main/resources/driver/chromedriver.exe");
        WebDriver driver = createWebDriver(device);
        String cdnUrl = "";
        try {
            driver.get(uri);
            cdnUrl = savePage(driver, url, "capture_" + System.currentTimeMillis() + ".png");
        } catch (Exception e) {
            logger.error("capture fail: [{},{}],{} !", uri ,device,e);
        } finally {
            driver.close();
            driver.quit();
        }
        PerfCounter.count("miui_ad_schedule_capture.execute.success", 1L, System.currentTimeMillis() - start);
        return cdnUrl;
    }

    private String savePage(WebDriver driver, String filePath, String fileName) throws IOException, InterruptedException, NoSuchAlgorithmException {
        JavascriptExecutor jsExecutor = (JavascriptExecutor) driver;
        TakesScreenshot takesScreenshot = (TakesScreenshot) driver;
        jsExecutor.executeScript("window.scrollTo(0,0)");
        BufferedImage imageOriginal = new BufferedImage(1, 1, BufferedImage.TYPE_INT_RGB);// 创建全屏截图
        int lastScroll = -1;
        int currentScroll = 0;
        Thread.sleep(1000);
        while (lastScroll != currentScroll) {
            byte[] bytesScroll = takesScreenshot.getScreenshotAs(OutputType.BYTES);
            BufferedImage imageScroll = ImageIO.read(new ByteArrayInputStream(bytesScroll));
            int screenHeight = imageScroll.getHeight();
            int screenWidth = imageScroll.getWidth();
            System.out.println("scree:" + screenHeight + ":" + screenWidth);

            BufferedImage combined = new BufferedImage(screenWidth, currentScroll + screenHeight, BufferedImage.TYPE_INT_RGB);
            Graphics g = combined.getGraphics();
            g.drawImage(imageOriginal, 0, 0, null);
            g.drawImage(imageScroll, 0, currentScroll, null);
            imageOriginal = combined;

            logger.info("lastScroll:" + lastScroll + "    currentScroll:" + currentScroll + "    screenHeight:" + screenHeight);
            int scrollTo = currentScroll + screenHeight;
            lastScroll = currentScroll;
            jsExecutor.executeScript("window.scrollTo(0," + scrollTo + ")");
            currentScroll = Double.valueOf(jsExecutor.executeScript("return document.documentElement.scrollTop").toString()).intValue();
            System.out.println(screenHeight + ":" + scrollTo);
            if (lastScroll > 5000) {
                break;
            }
            Thread.sleep(1000);
        }

        File path = new File(filePath);
        if (!path.exists() || !path.isDirectory()) {
            path.mkdirs();
        }
        File file = new File(path.getAbsolutePath() + File.separatorChar + fileName);
        ImageIO.write(imageOriginal, "png", file);
        String uploadPath = FileStorageUtils.upload(file.getAbsolutePath(), CHANNEL_ID);
        System.out.println("uploadPath:" + uploadPath);
        if(file.exists()){
            file.delete();
        }
        return uploadPath;
    }

(1) 安装chrome

sudo apt-get install libxss1 libappindicator1 libindicator7 #install dependency
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
sudo dpkg -i google-chrome*.deb  
sudo apt-get install -f

安装完成后测试：

google-chrome --headless --remote-debugging-port=9222 https://chromium.org --disable-gpu

这里是使用headless模式进行远程调试，ubuntu上大多没有gpu，所以--disable-gpu以免报错。之后使用另一个命令行访问本地的9222端口：

curl http://localhost:9222

能够看到调试信息应该就是安装成功了

(2) 下载chromedriver

chromedriver提供了操作chrome的api，是selenium控制chrome的桥梁。
可以到：https://sites.google.com/a/ch...。
下载并解压：

wget https://chromedriver.storage.googleapis.com/2.37/chromedriver_linux64.zip
unzip chromedriver_linux64.zip

Chrome与WebDriver的版本对应关系：

(3) 安装Xfvb

如果不安装Xfvb,java -jar xxx.jar 将报一下错误：

Starting ChromeDriver 2.30.477691 (6ee44a7247c639c0703f291d320bdf05c1531b57) on port 14103
Only local connections are allowed.
Exception in thread "main" org.openqa.selenium.WebDriverException: unknown error: Chrome failed to start: exited abnormally
  (Driver info: chromedriver=2.30.477691 (6ee44a7247c639c0703f291d320bdf05c1531b57),platform=Linux 4.9.15-x86_64-linode81 x86_64) (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 60.13 seconds
Build info: version: "unknown", revision: "unknown", time: "unknown"
System info: host: "localhost", ip: "127.0.0.1", os.name: "Linux", os.arch: "amd64", os.version: "4.9.15-x86_64-linode81", java.version: "1.8.0_131"
Driver info: driver.version: ChromeDriver
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    ...

安装方式：
a.安装Xvfb

sudo apt install Xvfb

b.启动Xvfb服务

Xvfb -ac :7 -screen 0 1280x1024x8 &

c.连接服务7

export DISPLAY=:7

关于Xvfb的具体应用请参考:https://www.x.org/archive/X11R7.6/doc/man/man1/Xvfb.1.xhtml

参考链接:
https://blog.csdn.net/codebat...
https://jiayi.space/post/zai-...

博客编写不易，如果觉得写得可以，请帮忙点个赞