{"id":1473,"date":"2014-10-21T09:10:52","date_gmt":"2014-10-21T00:10:52","guid":{"rendered":"http:\/\/blog.systemworks.co.jp\/?p=1473"},"modified":"2014-10-21T09:01:23","modified_gmt":"2014-10-21T00:01:23","slug":"infiniband%e3%81%ab%e3%82%88%e3%82%8bnfsrdma%e3%81%ae%e9%80%9f%e5%ba%a6%e6%a4%9c%e8%a8%bc","status":"publish","type":"post","link":"https:\/\/blog.systemworks.co.jp\/?p=1473","title":{"rendered":"InfiniBand\u306b\u3088\u308bNFS\/RDMA\u306e\u901f\u5ea6\u691c\u8a3c"},"content":{"rendered":"<p>NFS\/RDMA\u306f\u3001Infiniband\u306e\u9ad8\u901f\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u4e0a\u3067\u3001NFS\u30b5\u30fc\u30d0\uff0dNFS\u30af\u30e9\u30a4\u30a2\u30f3\u30c8\u9593\u306b\u304a\u3044\u3066\u3001\u30e1\u30e2\u30ea\u30c7\u30fc\u30bf\u306e\u9060\u9694\u8ee2\u9001(RDMA = Remote Direct Memory Access)\u3092\u884c\u3046\u6280\u8853\u3067\u3059\u3002<\/p>\n<p>InfiniBand\u30a2\u30c0\u30d7\u30bf\/\u30b9\u30a4\u30c3\u30c1\u3092\u5229\u7528\u3059\u308b\u3068\u3001\u9ad8\u901f\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u3092\u69cb\u7bc9\u3059\u308b\u3053\u3068\u304c\u51fa\u6765\u307e\u3059\u3002<br \/>\n\u3055\u3089\u306b\u3001RDMA (Remote Direct Memory Access) \u3082\u5229\u7528\u3059\u308b\u3053\u3068\u3067\u3001\u9ad8\u901f\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u4e0a\u306e\u30b3\u30f3\u30d4\u30e5\u30fc\u30bf\u9593\u3067\u30c7\u30fc\u30bf\u306e\u30e1\u30e2\u30ea\u30fc\u9593\u8ee2\u9001\u3092\u884c\u3046\u3053\u3068\u304c\u51fa\u6765\u307e\u3059\u3002RDMA\u3067\u306fCPU \u306e\u4ecb\u5165\u304c\u307b\u3068\u3093\u3069\u306a\u304f\u3001\u30b3\u30f3\u30d4\u30e5\u30fc\u30bf\u9593\u3067\u30e1\u30e2\u30ea\u30c7\u30fc\u30bf\u306e\u9060\u9694\u8ee2\u9001\u304c\u51fa\u6765\u308b\u305d\u3046\u3067\u3059\u3002\u3053\u308c\u3089\u3092NFS\u3067\u5229\u7528\u3059\u308b\u3068\u3001\u30c7\u30fc\u30bf\u8ee2\u9001\u901f\u5ea6\u30a2\u30c3\u30d7\u3092\u671f\u5f85\u51fa\u6765\u307e\u3059\u3002<\/p>\n<p>NFS\/RDMA\u3092\u5229\u7528\u3059\u308b\u5834\u5408\/\u5229\u7528\u3057\u306a\u3044\u5834\u5408\u3067\u3001\u3069\u306e\u7a0b\u5ea6\u9055\u3044\u304c\u3042\u308b\u306e\u304b\u691c\u8a3c\u3092\u884c\u3044\u307e\u3057\u305f\u3002<\/p>\n<p>NFS\/RDMA\u52d5\u4f5c\u306b\u306f\u3001\u30ab\u30fc\u30cd\u30eb\u30e2\u30b8\u30e5\u30fc\u30eb\u304c\u5fc5\u8981\u3067\u3001\u30e2\u30b8\u30e5\u30fc\u30ebsvcrdma\u3068xprtrdma\u306f\u3001Kernel 2.6.25\u4ee5\u964d\u304b\u3089\u5171\u306b\u30de\u30fc\u30b8\u3055\u308c\u3066\u3044\u307e\u3059\u3002Infiniband\u30c9\u30e9\u30a4\u30d0\u3092\u30a4\u30f3\u30b9\u30c8\u30fc\u30eb\u3057\u3066\u3001\u5fc5\u8981\u306a\u8a2d\u5b9a\u3092\u884c\u3048\u3070\u3001NFS\/RDMA\u3092\u5229\u7528\u3067\u304d\u307e\u3059\u3002\u4eca\u56de\u3001OFED\u306e\u30db\u30c3\u30c8\u30b9\u30bf\u30c3\u30af(\u30d0\u30fc\u30b8\u30e7\u30f33.12-1)\u3092\u4f7f\u7528\u3057\u307e\u3057\u305f\u3002<\/p>\n<p><strong>\u691c\u8a3c\u74b0\u5883<\/strong><\/p>\n<ul>\n<li>HCA\uff1a<a href=\"http:\/\/www.mellanox.com\/page\/infiniband_cards_overview\" target=\"_blank\">Mellanox MHQ19B-XTR<\/a> (QDR)<\/li>\n<li>CPU\uff1aXeon E5-2630V2 1\u3064\u642d\u8f09<\/li>\n<li>\u30de\u30b6\u30fc\u30dc\u30fc\u30c9\uff1a<a href=\"http:\/\/www.supermicro.com\/products\/motherboard\/Xeon\/C600\/X9DRi-F.cfm\" target=\"_blank\">Supermicro X9DRi-F<\/a><\/li>\n<li>OS\uff1aCentOS6.4 x86_64<\/li>\n<li>Infiniband\u30c9\u30e9\u30a4\u30d0\uff1a<br \/>\n<a href=\"https:\/\/www.openfabrics.org\/index.php\" target=\"_blank\">OFED<\/a>-3.12-1<\/li>\n<li>NFS\u30b5\u30fc\u30d0\u306e\u30b7\u30b9\u30c6\u30e0\u30e1\u30e2\u30ea\uff1a64GB(64GB\u30e1\u30e2\u30ea\u51851GB\u3092\u30b7\u30b9\u30c6\u30e0\u3067\u5229\u7528\u3057\u3001\u6b8b\u308a\u306ftmpfs\u3067\u4f7f\u7528\u3002tmpfs\u3092\/nfs\u3078\u30de\u30a6\u30f3\u30c8\u3002)<\/li>\n<li>NFS\u30af\u30e9\u30a4\u30a2\u30f3\u30c8\u306e\u30b7\u30b9\u30c6\u30e0\u30e1\u30e2\u30ea\uff1a1GB<\/li>\n<li>HCA\u30dd\u30fc\u30c8\u306bIP\u30a2\u30c9\u30ec\u30b9\u3092\u8a2d\u5b9a<br \/>\nNFS\u30b5\u30fc\u30d0IP\u30a2\u30c9\u30ec\u30b9 192.168.1.1<br \/>\nNFS\u30af\u30e9\u30a4\u30a2\u30f3 IP\u30a2\u30c9\u30ec\u30b9 192.168.1.2<\/li>\n<li>NFS\u30b5\u30fc\u30d0\uff0dNFS\u30af\u30e9\u30a4\u30a2\u30f3\u30c8\u9593\u306f\u4e00\u5bfe\u4e00\u3067\u63a5\u7d9a\u3057\u307e\u3057\u305f\u3002<\/li>\n<\/ul>\n<p><strong>\u8a2d\u5b9a<\/strong><\/p>\n<ol>\n<li>OFED\u3092NFS\u30b5\u30fc\u30d0\u3068NFS\u30af\u30e9\u30a4\u30a2\u30f3\u30c8\u3078\u30a4\u30f3\u30b9\u30c8\u30fc\u30eb<\/li>\n<li>NFS\u30b5\u30fc\u30d0\u306e\u8a2d\u5b9a<\/li>\n<table border=\"1\" cellspacing=\"0\" width=\"100%\">\n<tbody>\n<tr>\n<td width=\"100%\" valign=\"TOP\">\n# mount -t tmpfs -o size=63g tmpfs \/nfs<br \/>\n# vi \/etc\/fstab<br \/>\n\uff5e<br \/>\n\/nfs 192.168.1.0\/24(rw,sync,no_root_squash,fsid=0,insecure)<br \/>\n\uff5e\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<li>NFS\u30b5\u30fc\u30d0\u5074RDMA\u30e2\u30b8\u30e5\u30fc\u30eb\u306e\u52d5\u4f5c\u958b\u59cb<\/li>\n<table border=\"1\" cellspacing=\"0\" width=\"100%\">\n<tbody>\n<tr>\n<td width=\"100%\" valign=\"TOP\">\n# modprobe svcrdma<br \/>\n# service nfs start<br \/>\n# echo rdma 20049 > \/proc\/fs\/nfsd\/portlist\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<li>NFS\u30af\u30e9\u30a4\u30a2\u30f3\u30c8\u5074RDMA\u30e2\u30b8\u30e5\u30fc\u30eb\u306e\u52d5\u4f5c\u958b\u59cb\u3068\u30de\u30a6\u30f3\u30c8<\/li>\n<table border=\"1\" cellspacing=\"0\" width=\"100%\">\n<tbody>\n<tr>\n<td width=\"100%\" valign=\"TOP\">\n# modprobe xportrdma<br \/>\n# mount -o rdma,port=20049 192.168.1.1:\/nfs \/mnt\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/ol>\n<p><strong>\u30c6\u30b9\u30c8\u65b9\u6cd5<\/strong><\/p>\n<ul>\n<li>iozone\u3092\u5229\u7528\u3057\u30012GB\u30d5\u30a1\u30a4\u30eb(\u30b7\u30b9\u30c6\u30e0\u30e1\u30e2\u30ea\u306e2\u500d\u30b5\u30a4\u30ba\u30d5\u30a1\u30a4\u30eb)\u306e\u66f8\u304d\u51fa\u3057\/\u8aad\u307f\u51fa\u3057\u901f\u5ea6\u3092\u8a08\u6e2c\u3002<\/li>\n<table border=\"1\" cellspacing=\"0\" width=\"100%\">\n<tbody>\n<tr>\n<td width=\"100%\" valign=\"TOP\">\n# iozone -Ac -s 2g  -f \/mnt\/file\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>\u306a\u304a\u3001NFS\u30b5\u30fc\u30d0\u3067\u3001\/nfs\u3078\u30de\u30a6\u30f3\u30c8\u3057\u305ftmpfs\u306b\u5bfe\u3057\u3066\u3001iozone\u306b\u3088\u308a\u540c\u3058\u30d5\u30a1\u30a4\u30eb\u30b5\u30a4\u30ba\u3067\u901f\u5ea6\u3092\u8a08\u6e2c\u3059\u308b\u3068Write 3GB\/sec\u7a0b\u5ea6\u3001Read 6GB\/sec\u7a0b\u5ea6\u3067\u3057\u305f\u3002\n<\/ul>\n<p><strong>\u7d50\u679c<\/strong><\/p>\n<p>\u30ec\u30b3\u30fc\u30c9\u30b5\u30a4\u30ba4k\u301c16MB\u307e\u3067\u306e\u7d50\u679c\u306e\u5e73\u5747\u5024\u3092\u30b0\u30e9\u30d5\u5316\u3057\u307e\u3057\u305f\u3002(\u6a19\u6e96\u504f\u5dee\u306fNFS\/RDMA\u4f7f\u7528\u6642\u3001\u672a\u4f7f\u7528\u6642\u3068\u3082\u306bWrite\u306790\u304f\u3089\u3044\u3001Read\u306725\u304f\u3089\u3044\u3002)<\/p>\n<table border=\"0\" cellspacing=\"0\" width=\"100%\">\n<tbody>\n<tr>\n<td width=\"100%\" valign=\"TOP\">\n<a href=\"https:\/\/blog.systemworks.co.jp\/wp-content\/uploads\/2014\/10\/result1.jpg\"  target=\"_blank\"><img decoding=\"async\" src=\"https:\/\/blog.systemworks.co.jp\/wp-content\/uploads\/2014\/10\/result1.jpg\" alt=\"\" title=\"result1\" width=\"90%\" height=\"90%\" class=\"alignleft size-full wp-image-1507\" \/><\/a>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>Write\u306b\u306f\u3042\u307e\u308a\u9055\u3044\u304c\u3042\u308a\u307e\u305b\u3093\u304c\u3001Read\u3067\u306f\u500d\u4ee5\u4e0a\u306e\u5dee\u3092\u5f97\u3089\u308c\u307e\u3057\u305f\u3002<br \/>\nWrite\u3067\u3082\u3001\u5dee\u304c\u3082\u3063\u3068\u51fa\u3066\u6b32\u3057\u3044\u306e\u3067\u3059\u304c\u3001NFS\u8a2d\u5b9a\u30d5\u30a1\u30a4\u30eb\u306f\u521d\u671f\u8a2d\u5b9a\u306e\u307e\u307e\u3067\u3057\u305f\u306e\u3067\u3001\u30c1\u30e5\u30fc\u30cb\u30f3\u30b0\u3059\u308b\u3053\u3068\u306b\u3088\u308a\u3001\u7d50\u679c\u304c\u5909\u308f\u308b\u304b\u3082\u3057\u308c\u307e\u305b\u3093\u3002<\/p>\n","protected":false},"excerpt":{"rendered":"<p>NFS\/RDMA\u306f\u3001Infiniband\u306e\u9ad8\u901f\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u4e0a\u3067\u3001NFS\u30b5\u30fc\u30d0\uff0dNFS\u30af\u30e9\u30a4\u30a2\u30f3\u30c8\u9593\u306b\u304a\u3044\u3066\u3001\u30e1\u30e2\u30ea\u30c7\u30fc\u30bf\u306e\u9060\u9694\u8ee2\u9001(RDMA = Remote Direct Memory Access)\u3092\u884c\u3046\u6280\u8853\u3067\u3059\u3002  [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7],"tags":[],"class_list":["post-1473","post","type-post","status-publish","format-standard","hentry","category-7"],"_links":{"self":[{"href":"https:\/\/blog.systemworks.co.jp\/index.php?rest_route=\/wp\/v2\/posts\/1473","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.systemworks.co.jp\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.systemworks.co.jp\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.systemworks.co.jp\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.systemworks.co.jp\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1473"}],"version-history":[{"count":47,"href":"https:\/\/blog.systemworks.co.jp\/index.php?rest_route=\/wp\/v2\/posts\/1473\/revisions"}],"predecessor-version":[{"id":1475,"href":"https:\/\/blog.systemworks.co.jp\/index.php?rest_route=\/wp\/v2\/posts\/1473\/revisions\/1475"}],"wp:attachment":[{"href":"https:\/\/blog.systemworks.co.jp\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1473"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.systemworks.co.jp\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1473"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.systemworks.co.jp\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1473"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}